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The  Research  and  Technology 
Organisation  (RTO)  of  NATO 


ORGANIZATION 


RTO  is  the  single  focus  in  NATO  for  Defence  Research  and  Technology  activities.  Its  mission  is  to  conduct  and  promote 
co-operative  research  and  information  exchange.  The  objective  is  to  support  the  development  and  effective  use  of 
national  defence  research  and  technology  and  to  meet  the  military  needs  of  the  Alliance,  to  maintain  a  technological 
lead,  and  to  provide  advice  to  NATO  and  national  decision  makers.  The  RTO  performs  its  mission  with  the  support  of  an 
extensive  network  of  national  experts.  It  also  ensures  effective  co-ordination  with  other  NATO  bodies  involved  in  R&T 
activities. 


RTO  reports  both  to  the  Military  Committee  of  NATO  and  to  the  Conference  of  National  Armament  Directors. 
It  comprises  a  Research  and  Technology  Board  (RTB)  as  the  highest  level  of  national  representation  and  the  Research 
and  Technology  Agency  (RTA),  a  dedicated  staff  with  its  headquarters  in  Neuilly,  near  Paris,  France.  In  order  to 
facilitate  contacts  with  the  military  users  and  other  NATO  activities,  a  small  part  of  the  RTA  staff  is  located  in  NATO 
Headquarters  in  Brussels.  The  Brussels  staff  also  co-ordinates  RTO’s  co-operation  with  nations  in  Middle  and  Eastern 
Europe,  to  which  RTO  attaches  particular  importance  especially  as  working  together  in  the  field  of  research  is  one  of  the 
more  promising  areas  of  co-operation. 


The  total  spectrum  of  R&T  activities  is  covered  by  the  following  7 
•  AVT  Applied  Vehicle  Technology  Panel 
Human  Factors  and  Medicine  Panel 
Information  Systems  Technology  Panel 
NMSG  NATO  Modelling  and  Simulation  Group 
SAS  System  Analysis  and  Studies  Panel 

Systems  Concepts  and  Integration  Panel 
Sensors  and  Electronics  Technology  Panel 
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bodies: 


These  bodies  are  made  up  of  national  representatives  as  well  as  generally  recognised  ‘world  class’  scientists.  They  also 
provide  a  communication  link  to  military  users  and  other  NATO  bodies.  RTO’s  scientific  and  technological  work  is 
carried  out  by  Technical  Teams,  created  for  specific  activities  and  with  a  specific  duration.  Such  Technical  Teams  can 
organise  workshops,  symposia,  field  trials,  lecture  series  and  training  courses.  An  important  function  of  these  Technical 
Teams  is  to  ensure  the  continuity  of  the  expert  networks. 

RTO  builds  upon  earlier  co-operation  in  defence  research  and  technology  as  set-up  under  the  Advisory  Group  for 
Aerospace  Research  and  Development  (AGARD)  and  the  Defence  Research  Group  (DRG).  AGARD  and  the  DRG  share 
common  roots  in  that  they  were  both  established  at  the  initiative  of  Dr  Theodore  von  Karman,  a  leading  aerospace 
scientist,  who  early  on  recognised  the  importance  of  scientific  support  for  the  Allied  Armed  Forces.  RTO  is  capitalising 
on  these  common  roots  in  order  to  provide  the  Alliance  and  the  NATO  nations  with  a  strong  scientific  and  technological 
basis  that  will  guarantee  a  solid  base  for  the  future. 


The  content  of  this  publication  has  been  reproduced 
directly  from  material  supplied  by  RTO  or  the  authors. 


Published  September  2010 

Copyright  ©  RTO/NATO  2010 
All  Rights  Reserved 

ISBN  978-92-837-0113-2 
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Abbreviations,  Acronyms,  Definitions  and  Terms 


ACO 

ACT 

ACTD 

AFI 

AFMAN 

Allied  Command  Operations 

Allied  Command  Transformation 

Advanced  Concept  Technology  Demonstration 

Air  Force  Instruction  (USA) 

Air  Force  Manual  (USA) 

DoD 

DoDD 

DR 

Department  of  Defense  (USA) 

Department  of  Defense  Directive  (USA) 

Deficiency  Report  or  Deficiency  Reporting 

EPOW 

Experimentation  Programme  Of  Work 

GLP 

GMP 

Good  Laboratory  Practices 

Good  Manufacturing  Practices 

KPP 

KSA 

Key  Performance  Parameters 

Key  System  Attributes 

MOE 

MOP 

MTD 

Measure  Of  Effectiveness 

Measures  Of  Performance 

Medical  Technology  Demonstration 

NAMSA 

NATO 

NTO 

NATO  Maintenance  and  Supply  Agency 

North  Atlantic  Treaty  Organisation 

NATO  Test  Organisation 

OT  TC 
OT&E 
OTRR 

Operational  Test  Test  Concept 

Operational  Test  and  Evaluation 

Operational  Test  Readiness  Review 

SPOW 

Scientific  Program  Of  Work — Term  now  changed  to  EPOW 

T&E 

TEMP 

TRL 

Test  and  Evaluation 

Test  and  Evaluation  Master  Plan 

Technology  Readiness  Level 

DEFINITIONS/TERMS 

Advanced  Concept  Technology  Demonstration  -  A  demonstration  of  the  military  utility  of  a  significant  new 
technology  and  an  assessment  to  clearly  establish  operational  utility  and  system  integrity.  Within  the  NATO 
context  and  for  the  purposes  of  this  document,  this  term  has  been  modified  to  Medical  Technology 
Demonstration  (MTD). 

Capability  Based  Testing  -  A  mission-focused  methodology  of  verifying  that  a  capabilities  solution  will 
enable  operations  at  an  acceptable  level  of  risk.  Capabilities-oriented  evaluations  are  emphasized  throughout 
system  testing  in  addition  to  traditional  evaluations  of  system  performance  measured  against  specification-like 
requirements.  It  requires  understanding  Concept  of  Operations  and  involves  developing  T&E  strategies  and 
plans  to  determine  whether  a  capability  solution  option  merits  fielding. 
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Critical  Operational  Issue  (COI) 

1)  Operational  effectiveness  and  operational  suitability  issues  (not  parameters,  objectives,  or  thresholds)  that 
must  be  examined  during  operational  testing  to  determine  the  system’s  capability  to  perform  its  mission. 

2)  A  key  question  that  must  be  examined  in  operational  test  and  evaluation  to  determine  the  system’s 
capability  to  perform  its  mission.  Testers  normally  phrase  a  COI  as  a  question  to  be  answered  in  evaluating 
a  system’s  operational  effectiveness  or  suitability. 

Compatibility  -  The  suitability  of  products,  processes  or  services  for  use  together  under  specific  conditions  to 
fulfill  relevant  requirements  without  causing  unacceptable  interactions.  (AAP-6) 

Deficiency  Report  (DR)  -  The  report  used  to  identify,  document,  and  track  system  deficiency  or  enhancement 
data  while  a  system  is  in  advanced  development,  operational  test,  or  operational  transition. 

-  Category  I  -  DRs  are  those  which  could  cause  death,  severe  injury,  severe  occupational  illness,  major 
loss  or  damage,  or  directly  restrict  combat  or  operational  readiness  if  left  uncorrected. 

-  Category  II  -  DRs  are  those  which  do  not  meet  the  criteria  of  a  Cat  I  DR.  They  are  attributable  to 
errors  in  workmanship,  non-conformance  to  specifications,  drawing  standards,  or  other  technical 
requirements;  or  identify  a  problem  for  potential  improvement  or  enhancement. 

-  Enhancements  are  a  type  of  Category  II  DR  which  identifies  conditions  that  complement,  but  are  not 
absolutely  required  for  successful  mission  accomplishment.  The  recommended  condition,  if  incorporated, 
will  improve  a  system’s  operational  effectiveness  or  suitability. 

Key  Performance  Parameters  -  Required  system  capabilities.  If  not  met,  the  device/system  fails  the  test  and 
evaluation  and  is  deemed  non-deployable. 

Key  System  Attributes  -  Capabilities  that  are  desired,  but  not  required  for  deployment/utilization. 

Measures  Of  Performance  (MOP)  —  Testable  and  measurable  attributes  of  the  performance  which  can  be 
evaluated  by  a  T&E  process. 

Operational  Test  and  Evaluation  (OT&E) 

1)  The  field  test,  under  realistic  combat  conditions,  of  any  item  of  (or  key  component  of)  equipment  for  the 
purpose  of  determining  the  effectiveness  and  suitability  of  the  equipment  for  use  in  combat  by  typical  military 
users;  and  the  evaluation  of  the  results  of  such  test. 

2)  Testing  and  evaluation  conducted  in  as  realistic  an  operational  environment  as  possible  to  estimate  the 
prospective  system’s  operational  effectiveness  and  operational  suitability.  In  addition,  OT&E  provides 
information  on  organization,  personnel  requirements,  doctrine,  and  tactics.  It  may  also  provide  data  to 
support  or  verify  material  in  operating  instructions,  publications,  and  handbooks. 

Operational  Testing  -  A  generic  term  describing  the  test  and  evaluation  options  and  levels  of  effort  available 
to  an  operational  test  organization. 

Oversight  -  Senior  executive-level  monitoring  and  review  of  programs  to  ensure  compliance  with  policy  and 
attainment  of  broad  program  goals. 

Test  and  Evaluation  (T&E)  -  The  act  of  generating  empirical  data  during  the  research,  development  or 
sustainment  of  systems,  and  the  creation  of  information  through  analysis  that  is  useful  to  technical  personnel 
and  decision  makers  for  reducing  design  and  acquisition  risks.  The  process  by  which  systems  are  measured 
against  requirements  and  specifications,  and  the  results  analyzed  so  as  to  gauge  progress  and  provide  feedback. 
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Test  and  Evaluation  Master  Plan  (TEMP)  -  Documents  the  overall  structure  and  objectives  of  the  T&E 
program.  It  provides  a  framework  within  which  to  generate  detailed  T&E  plans  and  it  documents  schedule  and 
resource  implications  associated  with  the  T&E  program.  The  TEMP  identifies  the  necessary  developmental, 
operational,  and  live  test  activities.  It  relates  program  schedule,  test  management  strategy  and  structure, 
and  required  resources  to:  COIs;  critical  technical  parameters;  objectives  and  thresholds  documented  in  the 
requirements  document. 

Test  and  Evaluation  Organization  -  Any  organization  whose  designated  mission  includes  test  and  evaluation. 
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Development  of  an  Assessment  Methodology 
for  Demonstrating  Usability,  Technical 
Maturity,  and  Operational  Benefits 
of  Advanced  Medical  Technology 

(RT  O-TR-HFM- 130) 

Executive  Summary 

Many  nations,  both  within  the  Alliance  and  those  of  the  Partnership  for  Peace,  have  in  recent  years 
developed  various  modalities  of  advanced  medical  technologies  for  operational  field  use.  These  range  from 
classic  teleconsultation  through  biosurveillance  systems  and  distance  education  to  infonnation  management 
systems  for  maintaining  medical  awareness  of  the  battlefield,  and  most  recently  the  concept  of  the  use  of 
Unmanned  Aerial  Vehicles  (UAVs)  for  casualty  evacuation  (CASEVAC).  Some  of  these  groups  of 
technologies  may  include  real-time  soldier/patient  status  monitoring,  patient  tracking,  medical  logistics  status 
monitoring,  and  epidemiological  reporting.  The  Allied  Command  Operations  Medical  Advisor  has 
determined  that  these  advanced  technologies  can  potentially  be  of  great  benefit  as  a  medical  force  multiplier, 
and  has  requested  that  telemedicine  in  this  broader  sense  be  incoiporated  into  the  medical  support  structures 
for  the  NATO  Response  Force.  The  importance  of  these  developments  is  clearly  shown  by  the  continuing 
RTO  efforts  to  identify  and  evaluate  such  technology  (e.g.,  RTG-182,  “Use  of  Advanced  Technologies  and 
New  Procedures  in  Medical  Field  Operations”  and  RTG-184,  “Safe  Ride  Standards  for  Patient  Evacuation 
Using  Unmanned  Aerial  Vehicles  (UAVs)”).  However,  to  date  there  is  no  standardized  approach  to  evaluate 
under  test  circumstances  the  operational  benefit  and  functionality  of  these  new  technologies. 

Many  of  these  technologies  are  being  fielded  at  the  national  level,  based  on  a  national  “gestalt”  that  potential 
benefits  of  their  use  are  obvious.  Unfortunately,  adoption  of  any  particular  technology  or  telemedicine 
modality  for  Alliance  support  is  hindered  by  the  lack  of  any  standardized  NATO  assessment  mechanism 
which  can  evaluate  and  assess  new  medical  technologies  for  technical  maturity,  usability  (in  the  classic 
human  factors  sense),  and  benefits  to  the  patient,  the  clinician,  and  the  operational  commander.  Development 
of  such  an  evaluation  mechanism  could  lead  to  increased  deployment  of  such  advanced  medical 
technologies,  as  well  as  increased  interoperability.  Widespread  deployment  of  such  technologies  during 
multi-national  NATO  operations  would  have  obvious  potential  benefit  in  reducing  morbidity  and  mortality, 
allowing  a  reduced  medical  “footprint”  in  the  operational  area,  and  in  allowing  the  NATO  Theatre  Surgeon 
and  the  Operational  Commanders  to  maintain  an  improved  understanding  of  the  medical  status  of  their 
deployed  forces  and  the  deployed  medical  support  structures.  The  availability  of  a  standardized  assessment 
and  evaluation  methodology  could  significantly  assist  in  the  decision-making  as  to  whether  or  not  to  field 
such  new  technology  in  the  multi-national  environment. 

RTG-130  has  examined  the  various  mechanisms  currently  in  use  by  various  nations  to  test  the  field  usability 
of  such  devices,  and  has  carefully  examined  the  world  literature  on  Health  Technology  Assessment  and 
Medical  Equipment  Testing.  We  have  also  analyzed  the  current  exercise  and  material-testing  programs  of 
NATO  to  determine  how  such  testing  could  potentially  be  inserted  into  currently-existing  programs. 
A  schema  for  using  the  concept  of  Technology  Readiness  Levels  (TRLs)  in  the  evaluation  of  biomedical 
developments  has  been  developed  and  recommended  for  formal  NATO  adoption  and  use.  Finally, 
a  proposed  test  procedure/process  for  evaluation  and  testing  of  future  such  developments  for  use  within  the 
NATO  multi-national  environment  has  been  developed,  which  recommends  assignment  of  this  task  to  the 
Medical  Branch  at  Allied  Command  Transformation  (ACT). 
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Developpement  d’une  methodologie  d’evaluation 
permettant  de  demontrer  la  facilite  d’utilisation, 
la  maturite  technique  et  les  avantages 
operationnels  des  technologies 
medicales  evoluees 

(RT  O-TR-HFM- 130) 


Synthese 

Nombre  de  nations,  que  ce  soit  au  sein  de  l’Alliance  ou  dans  le  cadre  du  Partenariat  pour  la  paix,  ont 
developpe  au  cours  des  demieres  annees  diverses  modalites  relatives  aux  technologies  medicales  evoluees  et 
a  leur  utilisation  sur  le  theatre  des  operations.  Elies  varient  de  la  teleconsultation  classique  via  les  systemes 
de  bio-surveillance  et  l’enseignement  a  distance,  aux  systemes  de  gestion  de  l’information  visant  a  maintenir 
une  vigilance  medicale  sur  le  champ  de  bataille,  en  passant  plus  recemment  par  le  concept  d’utilisation  de 
Vehicules  aeriens  sans  pilote  (UAV)  pour  l’evacuation  des  blesses  (CASEVAC).  Certaines  de  ces 
technologies  peuvent  inclure  en  temps  reel  le  controle  de  l’etat  du  combattant/patient,  la  localisation  du 
patient,  le  controle  de  l’etat  de  la  logistique  medicale,  ou  encore  l’etablissement  de  rapports 
epidemiologiques.  Le  Conseiller  medical  du  Commandement  allie  pour  les  Operations  a  conclu  que  ces 
technologies  evoluees  pourraient  apporter  un  benefice  considerable  en  demultipliant  les  forces  medicales, 
et  a  demande  que  la  telemedecine  dans  son  acception  la  plus  large  soit  incorporee  dans  les  structures  de 
soutien  medical  de  la  Force  de  reaction  de  l’OTAN.  L’importance  de  ces  developpements  est  clairement 
demontree  par  la  poursuite  des  efforts  de  la  RTO  en  vue  d’ identifier  et  d’evaluer  ces  technologies 
(RTG-182  :  «  Utilisation  de  technologies  evoluees  et  de  nouvelles  procedures  dans  les  operations  medicales 
de  campagne  »,  RTG-184  :  «  Normes  de  securite  pour  l’evacuation  des  patients  a  l’aide  de  Vehicules  aeriens 
sans  pilote  (UAV)  »,  etc.).  Toutefois,  il  n’existe  a  ce  jour  aucune  approche  standardisee  pennettant  d’evaluer 
par  des  essais  les  benefices  operationnels  et  la  fonctionnalite  de  ces  nouvelles  technologies. 

Plusieurs  de  ces  technologies  sont  utilisees  sur  le  terrain  au  niveau  national,  en  se  fondant  sur  l’a  priori  selon 
lequel  les  benefices  potentiels  apportes  par  leur  mise  en  oeuvre  sont  evidents.  Malheureusement,  1’ absence 
d’un  mecanisme  d’evaluation  standardise  de  l’OTAN,  capable  de  determiner  la  maturite  technique,  la  facilite 
d’utilisation  (au  sens  classique  des  facteurs  humains)  et  les  benefices  de  ces  nouvelles  technologies 
medicales  pour  le  patient,  le  clinicien  et  le  commandement  operationnel,  empeche  l’adoption  d’une 
technologie  ou  d’une  modalite  de  telemedecine  particuliere  pour  le  soutien  de  l’Alliance.  Le  developpement 
d’un  tel  mecanisme  d’evaluation  pourrait  permettre  d’accroitre  le  deployment  de  ces  technologies  medicales 
evoluees,  ainsi  que  l’interoperabilite.  Le  deployment  generalise  de  ces  technologies  lors  d’ operations 
multinationales  de  l’OTAN  genererait  un  benefice  potentiel  evident,  permettant  de  reduire  la  morbidity  et  la 
mortalite,  de  limiter  «  l’empreinte  »  medicale  dans  la  zone  operationnelle,  et  procurant  au  Medecin-chef  du 
theatre  et  aux  Commandants  operationnels  de  l’OTAN  une  meilleure  connaissance  de  l’etat  medical  de  leurs 
forces  et  des  structures  de  soutien  medical  deployees.  La  mise  a  disposition  d’une  methodologie  d’evaluation 
standardisee  pourrait  faciliter  considerablement  la  prise  de  decision  quant  a  l’utilisation  ou  la  non-utilisation 
de  ces  nouvelles  technologies  dans  un  environnement  multinational. 

Le  RTG-130  a  examine  les  divers  mecanismes  actuellement  mis  en  oeuvre  par  plusieurs  nations,  en  vue  de 
tester  la  facilite  d’utilisation  de  ces  dispositifs  sur  le  terrain;  il  a  en  outre  soigneusement  etudie  la 
documentation  internationale  disponible  sur  T  evaluation  des  technologies  medicales  et  les  tests  realises 
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sur  le  materiel  medical.  Nous  avons  egalement  analyse  les  programmes  actuels  de  l’OTAN  relatifs  a 
l’essai  du  materiel  et  aux  exercices,  en  vue  d’etablir  dans  quelle  mesure  de  tels  essais  pourraient  etre 
inclus  dans  les  programmes  deja  existants.  Un  schema  d’utilisation  du  concept  de  Niveaux  de  maturite 
technologique  (TRL)  lors  de  T  evaluation  de  developpements  biomedicaux  a  ete  consu  et  recommande 
pour  adoption  formelle  et  utilisation  par  l’OTAN.  Enfin,  une  proposition  de  procedure  d’essai/processus 
devaluation  et  de  test  des  futurs  developpements,  a  utiliser  dans  l’environnement  international  de 
l’OTAN,  a  ete  presentee,  recommandant  l’attribution  de  cette  tache  a  la  branche  medicale  du 
Commandement  allie  Transformation  (ACT). 
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Chapter  1  -  INTRODUCTION 


ORGANIZATION 


1.1  ESTABLISHMENT  OF  THE  RTG 

In  2004,  the  Human  Factors  and  Medicine  (HFM)  Panel  of  the  NATO  Research  and  Technology  Organization 
(RTO)  established  an  Exploratory  Team,  HFM/ET-051,  with  the  mission  of  examining  the  topic  “Applications 
and  Assessments  for  Telemedicine  (TMED)  Support  to  the  NATO  Response  Force  (NRF)”.  The  justification 
for  this  ET  was  that  full-spectrum  operations  of  NATO  forces  (e.g.,  the  “NATO  Response  Force”  (NRF)) 
must  be  supported  and  sustained  by  the  most  modem  military  medical  systems  and  technologies. 
The  emergence  of  “telemedicine”  (TMED)  as  both  an  operational  domain  and  a  research  and  development 
(R&D)  discipline  is  currently  being  exploited  for  modernization  of  NATO  medical  support  capabilities 
through  various  activities  of  the  NATO  Military  Committee  (MC)  -  the  COMEDS  TMED  Expert  Team  which 
is  heavily  involved  in  developing  standards  for  TMED  within  the  Multi-National  Medical  Community,  and  in 
identifying  required  changes  to  NATO  policy  and  doctrine  to  incorporate  TMED  into  NATO  operational 
medical  doctrine. 

Throughout  2005,  this  ET  examined  the  issues,  and  coordinated  with  numerous  other  NATO  bodies  to 
determine  the  most  appropriate  way  ahead  toward  resolution  of  this  task.  Their  final  recommendation  was  that 
two  separate  RTG  Task  Groups  be  established,  one  of  which  would  examine  the  concept  of  Technological 
Readiness  Levels,  field  experimentation  and  evaluation  of  advanced  medical  technologies  within  the  NATO 
multi-national  environment.  It  was  believed  that  in  order  to  achieve  an  understanding  of  TMED  in  the  NATO 
operational  context,  a  HFM  activity  was  needed  in  order  to  discuss  assessments  of  TMED  technology  TRLs, 
to  survey  of  the  readiness  of  NATO  support  areas  to  deploy  TMED  technology,  and  to  determine  appropriate 
opportunities  to  conduct  cooperative  technology  demonstrations.  This  recommendation  was  accepted  by  the 
HFM  Panel  and  the  RTB,  and  resulted  in  the  formation  of  the  current  RTG-130.  The  other  recommended  RTG 
was  expected  to  look  into  the  concept  of  testing  and  evaluation  of  embedded  algorithms  within  medical 
systems  and  devices,  and  was  subsequently  organized  as  RTG-131,  which  is  not  a  subject  of  this  report. 

RTG-130  was  formed  with  a  mandate  to  look  at  the  entire  spectrum  of  advanced  medical  technology  and  its 
evaluation  rather  than  being  restricted  solely  to  Telemedicine  as  was  originally  envisaged.  The  implementation 
of  the  RTG  was  delayed  for  two  years,  for  multiple  reasons,  but  was  finally  able  to  start  work  on  this  topic  in  late 
October  of  2006.  Due  to  its  late  start,  a  lifespan  extension  until  October  2009  was  approved  by  the  RTB. 
In  carrying  out  its  mandate,  the  RTG  met  four  times  as  a  committee  of  the  whole,  with  additional  work  being 
carried  out  between  meetings. 

To  support  the  development  of  standards  and  doctrine,  there  is  a  need  to  understand  the  maturity  and  efficacy  of 
those  technologies  that  enable  the  effective  provision  of  field  medical  care  and  to  assess  critical  medical 
technologies  using  the  “Technology  Readiness  Level”  (TRL)  assessment  methodology.  Selected  “Cooperative 
Demonstrations  of  Technology”  (CDT)  for  support  to  the  NRF  and  other  NATO  operational  scenarios  (such  as 
ISAF)  can  serve  to  guide  the  technology  modernization  path  for  the  use  of  advanced  technologies  in  operational 
medicine.  The  importance  of  this  effort  is  demonstrated  by  the  fact  that  within  the  past  year  (2009)  an  agreement 
has  been  reached  between  SHAPE  and  the  U.S.  Army  Medical  Command  to  allow  NATO-led  forces  in 
Afghanistan  to  utilize  the  clinical  consultation  services  of  the  U.S.  Army’s  deployed  “AKO”  Teleconsultation 
System,  as  a  field  test  to  demonstrate  multi-national  feasibility,  but  without  any  formal  NATO  assessment 
process  or  system.  Establishment  of  an  evaluation  process  as  recommended  in  this  Technical  Report  would 
materially  assist  COMEDS,  ACT,  and  ACO  in  determining  which  new  medical  technologies  could  effectively 
enhance  NATO  multi-national  medical  care. 
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RTG-130  has  based  its  work  on  the  following. 


1.2  BACKGROUND  AND  JUSTIFICATION 

Many  nations,  both  within  the  Alliance  and  within  the  Partnership  for  Peace  (PfP)  program,  have  in  recent 
years  developed  various  modalities  of  advanced  medical  technologies  for  operational  field  use.  These  range 
from  classic  teleconsultation  through  biosurveillance  systems  and  distance  education  to  information 
management  systems  for  maintaining  medical  awareness  of  the  battlefield.  This  latter  group  of  technologies 
may  include  real-time  soldier/patient  status  monitoring,  patient  tracking,  medical  logistics  status  monitoring, 
and  epidemiological  reporting.  One  of  the  most  recent  proposals  is  for  the  use  of  unmanned  aerial  vehicles  as 
Casualty  Evacuation  (CASEVAC)  vehicles,  and  the  required  development  of  “safe  ride  standards”  for  such 
aircraft.  Such  developments  are  currently  the  subject  of  evaluation  by  RTG-182  and  RTG-184,  among  others. 
The  Allied  Command  Operations  Medical  Advisor  has  determined  that  these  advanced  technologies  can 
potentially  be  of  great  benefit  as  a  medical  force  multiplier,  and  has  requested  that  Telemedicine  in  its 
broadest  sense  be  incorporated  into  the  medical  support  structures  for  the  NATO  Response  Force  and  other 
NATO  deployed  and  deployable  forces.  These  advanced  medical  technologies  are  widely  seen  as  having 
potential  benefit  to  patients,  clinicians,  and  commanders  in  multi-national  operations  in  a  field  setting.  Many 
of  these  technologies  are  being  fielded  at  the  national  level,  based  on  a  national  “gestalt”  that  potential 
benefits  of  their  use  are  obvious,  or  based  on  national  testing  programs.  Unfortunately,  adoption  of  any 
particular  technology  or  telemedicine  modality  for  Alliance  operational  support  is  hindered  by  the  lack  of  any 
standardized  NATO  assessment  mechanism  which  can  evaluate  and  assess  new  medical  technologies  for 
technical  maturity,  usability  (in  the  classic  human  factors  sense),  and  benefits  to  the  patient,  the  clinician, 
and  the  operational  commander.  Development  of  such  an  evaluation  mechanism  could  lead  to  increased 
deployment  of  such  advanced  medical  technologies,  as  well  as  increased  interoperability,  widespread 
deployment  of  such  technologies  during  NATO  multi-national  operations  would  have  obvious  potential 
benefit  in  reducing  morbidity  and  mortality,  allowing  a  reduced  medical  “footprint”  in  the  operational  area, 
and  in  allowing  the  NATO  Theatre  Surgeon  and  the  Operational  Commanders  to  maintain  an  improved 
understanding  of  the  medical  status  of  their  deployed  forces  and  the  deployed  medical  support  structures. 
The  availability  of  a  standardized  assessment  and  evaluation  methodology  could  significantly  assist  in  the 
decision-making  as  to  whether  or  not  to  field  such  new  technology  in  the  multi-national  environment. 


1.3  OBJECTIVE(S) 

The  Technical  Group  was  assigned  to  carry  out  research  aimed  at  the  evaluation  of  current  and  proposed  test 
methods  to  determine  technical  maturity,  usability,  and  benefit  of  advanced  medical  technology  (including  both 
hardware  and  embedded  software).  A  large  number  of  technology  assessment  concepts  were  reviewed,  including 
the  general  concept  of  Health  Technology  Assessment  (HTA),  which  was  detennined  to  not  be  suitable  in  its 
entirety  for  NATO  use.  We  discovered  that  few  of  our  nations  have  specific  programs  designed  to  perform 
standardized  testing  and  evaluation  on  advanced  medical  technology  in  the  field  environment.  Frequently, 
it  appears  that  only  local  testing  by  potential  future  users  is  carried  out  prior  to  purchase  or  adoption,  rather  than 
any  type  of  a  centralized  Test  and  Evaluation  program.  We  evaluated  these  existing  programs  from  as  many 
nations  as  possible.  After  evaluation  of  these  methods,  a  suitable  “standardizable”  method  was  designed  for 
possible  NATO  standardization.  We  propose  that  this  selected  method  should  be  tested  in  the  field  during  a 
Cooperative  Technology  Demonstration,  which  could  carry  out  a  field  tri  a  1  /d c m o n s trat i o n/as scs s men  t  of  various 
devices  usable  at  Roles  1/2,  using  a  scenario  to  be  approved  by  the  ACO/ACT  Medical  Advisors.  If  successful, 
the  proposed  standardized  methodology  may  be  considered  for  publication  as  a  NATO  standardization 
document. 
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1.4  TOPICS  COVERED 

The  Task  Group  carried  out  a  historical  review  of  advanced  medical  technology  test  and  evaluation  procedures. 
It  also  discussed: 

•  Operational  issues; 

•  Functional  issues; 

•  Human  factors  usability  issues; 

•  NATO  requirements; 

•  Operational  test  and  evaluation;  and 

•  Standardization. 

Health  Technology  Assessment  (HTA),  and  all  its  ramifications  were  also  discussed  in  detail. 


1.5  DELIVERABLE  AND/OR  END  PRODUCTS 

The  group  has  produced: 

•  A  set  of  biomedical  Technology  Readiness  Level  (TRL)  definitions  which  are  recommended  for 
NATO  adoption  to  guide  future  considerations  for  equipment  development  and  acceptance; 

•  A  recommendation  for  identifying  a  NATO  body  which  could  be  tasked  with  overall  responsibility 
for  medical  equipment  Test  and  Evaluation  (T&E); 

•  A  standardized  NATO  test  and  evaluation  procedure  for  Advanced  Medical  Technology  proposed  for 
use  in  the  NATO  multi-national  medical  environment;  and 

•  Several  guidelines  for  use  in  evaluation  of  the  results  of  such  testing. 
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2.1  THE  NEED  FOR  TECHNOLOGY  ASSESSMENT  OF  MEDICAL 
TECHNOLOGIES  WITHIN  NATO 

Changes  in  NATO  policies  and  doctrine  highlight  the  many  changes  faced  by  the  Alliance  in  recent  years. 
The  Alliance  has  made  numerous  strides  in  adapting  to  its  new  role  in  the  world,  and  medical  support  is  an 
integral  part  of  this  new  situation.  Whereas  in  the  past  medical  support  was  considered  a  strictly  national 
responsibility,  it  is  now  considered  a  shared  responsibility  between  the  contributing  nations  and  the  NATO 
Commander1’2.  At  the  same  time,  the  operational  concepts  have  changed  to  the  extent  that  no  longer  is  it 
assumed  that  any  one  nation  will  provide  the  full  panoply  of  Role  1  through  Role  4  medical  support  to  any 
operation.  Current  NATO  doctrine  for  operational  medical  support  envisions  many  changes  in  future  medical 
support  structures,  including  true  multi-national  medical  units,  role  specialization,  host  nation  support, 
and  increased  requirements  for  interoperability  among  and  between  national  medical  support  structures3. 

Thus,  the  old  medical  operational  concept  in  which  a  nation  really  only  had  to  be  primarily  concerned  with 
support  of  its  own  troops  by  its  own  national  medical  system,  with  the  consequent  “simple”  requirement  for 
interoperability  only  within  its  own  system,  has  evolved.  Accordingly,  it  is  critical  that  any  nation  desiring  to 
deploy  new  or  advanced  medical  technology  to  support  NATO  operations  ensure  that  its  systems  are  usable 
by,  and  are  interoperable  with,  those  of  other  nations. 

This  change  from  totally  nationally-provided  medical  services  to  joint  responsibility  has  been  reflected  not 
only  in  doctrine,  but  also  in  organizational  structure.  The  Committee  of  the  Chiefs  of  the  Military  Medical 
Services  in  NATO  (COMEDS  -  The  Alliance’s  senior  medical  body),  the  NATO  Standardization  Agency, 
and  the  Joint  Medical  Committee  have  in  recent  years  changed  their  organizational  structures,  interrelationships, 
and  working  practices  to  reflect  this  new  reality. 

However,  the  nations  still  retain  the  responsibility  for  maintaining,  training,  and  equipping  their  own  military 
medical  forces.  At  the  present  time,  NATO  neither  purchases  nor  develops  medical  technology  with  the 
exception  of  the  NATO  Maintenance  and  Supply  Agency’s  (NAMSA)  current  efforts  in  hospital  procurement 
for  Afghanistan.  Development  and  procurement  of  medical  materiel  is  still  a  national  responsibility.  Thus,  it  is 
the  nations  which  have  always  decided  if  new  medical  technology  is  to  be  adopted  for  use  within  their  forces, 
often  without  full  consideration  of  the  impact  of  this  change  upon  NATO  medical  support.  An  old  quip  is  that 
the  only  medical  equipment  items  that  NATO  owns  as  an  alliance  are  the  first  aid  kits  on  their  AWACS 
aircraft.  This  is  not  totally  unfounded,  as  it  remains  the  nations  which  maintain  the  responsibility  for 
development,  procurement,  and  stocking  of  medical  materiel.  This  national  responsibility  in  the  logistics  field 
implies  that  the  nations,  not  NATO,  will  remain  primarily  responsible  for  the  development  and  fielding  of 
new  medical  support  structures  and  the  new  technologies  which  are  incorporated  therein.  This  concept  may 
need  to  change  as  there  is  recognized  to  be  more  need  for  standardized  NATO  medical  support  structures, 
which  may  be  reflected  in  future  capability  packages,  etc.  The  first  steps  toward  a  true  alliance  capability  in 
this  regard  might  be  reflected  in  the  early  2007  creation  of  a  medical  cell  at  the  NAMSA  and  the  currently- 
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proposed  multi-national  medical  facility  at  Camp  Bastion,  Afghanistan.  However,  until  such  a  change  is 
formally  made,  the  NATO  interest  in  new  fielded  medical  devices  and  technology  is  to  ensure  that  these 
developments: 

1)  Enhance  the  overall  medical  support  posture  of  the  Alliance; 

2)  Perform  as  intended  in  the  multi-national  field  environment; 

3)  Can  be  effectively  incorporated  into  NATO  medical  doctrine;  and 

4)  Are  interoperable  with  other  national  medical  systems  to  the  extent  needed  to  enhance  multi-national 
medical  care. 

At  the  same  time  as  we  are  seeing  these  organizational  and  operational  changes,  changes  in  medical 
technology  are  occurring  by  leaps  and  bounds.  Medical  technology  is  changing  rapidly  in  many  key  areas,  and 
the  speed  of  development  of  advanced  medical  technology  has  accelerated.  Last  year’s  technology  may  not 
provide  optimum  medical  support  on  tomorrow’s  battlefield,  and  currently  critical  technologies  may  either 
become  obsolescent  or  develop  rapidly.  Keeping  pace  with  technology  and  ensuring  that  new  medical 
equipment  proposed  for  use  within  the  Alliance  meets  operational  demands  will  be  challenging  in  the  21st 
century.  Taking  full  advantage  of  these  advances  in  medical  technology  to  provide  operationally  relevant 
medical  care  is  a  critical  aspect  of  the  transformation  of  the  Alliance.  To  take  full  advantage  of  these  advances 
in  medical  technology,  NATO  must  place  the  best  possible  medical  technology  in  the  hands  of  the  medical 
personnel  who  will  support  future  military  operations,  and  must  ensure  that  it  operates  as  intended,  that  it 
supports  current  and  evolving  NATO  medical  support  doctrine,  and  that  it  is  usable  by  personnel  and 
organizations  other  than  those  of  the  developer  nations.  These  demands  imply  that  there  is  a  requirement  for 
operational  testing  of  new  medical  technologies  at  the  NATO  level  before  new  national  equipment  should  be 
accepted  for  NATO  use  in  multi-national  operations.  It  is  NATO  as  an  Alliance  which  needs  to  test  the 
equipment,  not  for  simple  medical  function,  but  for  its  usability  and  adaptability  within  the  NATO  doctrinal 
structure. 

Many  if  not  most  of  the  critical  advanced  medical  technologies  are  developed  and  produced  by  commercial 
industries  rather  than  defence-related  or  national  facilities,  and  therefore  may  be  introduced  on  the  commercial 
market  and  purchased  by  the  nations  without  a  full  consideration  of  how  (or  whether)  they  could  be  used  in 
the  NATO  multi-national  environment.  The  different  levels  of  training,  language  skills,  and  technological 
know-how  which  may  be  found  in  different  national  contributions  to  a  NATO  operation  are  not  necessarily 
recognized  by  the  developers  or  by  national  medical  authorities.  It  is  therefore  critical  that  NATO  develop  a 
policy  which  can  address  the  issues  of  usability  and  compatibility  with  current  and  developing  doctrine  prior 
to  accepting  new  medical  technologies. 

We  consider  that  it  will  continue  to  be  a  primarily  national  responsibility  to  leverage  the  best  medical 
technology  available,  to  rapidly  transition  this  technology  into  new  medical  systems  and  force  structures,  and 
to  propose  its  use  within  the  NATO  environment.  However,  there  is  a  distinct  need  for  the  Alliance  to  become 
involved  in  determining  the  utility  of  that  technology  in  supporting  current  and  future  medical  doctrine, 
its  usability  in  the  multi-national  environment,  and  its  potential  human  factors  issues  when  users  come  from 
many  Allied  and  PfP  nations  with  differing  levels  of  medical  technological  background.  Whether  or  not  future 
doctrine  encompasses  actual  NATO  procurement  and  stockage  of  medical  equipment  and  supplies,  or  whether 
this  provision  remains  a  strictly  national  responsibility,  there  is  a  requirement  that  such  equipment  be  found  to 
be:  suitable  for  its  proposed  use;  usable  by  all  NATO  and  PfP  medical  personnel  who  might  have  to  operate, 
maintain,  or  repair  it;  and  coherent  with  NATO  medical  doctrine. 
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Before  a  new  system  is  fielded,  potential  operational  users  should  participate  in  testing  and  evaluating  the 
operation  of  the  system  to  ensure  that  the  new  system  is  effective  when  used  under  realistic  conditions  and 
will  meet  the  required  operational  need.  Thus,  there  is  a  perceived  need  for  a  NATO  Test  and  Evaluation 
(T&E)  policy  for  new  medical  equipment  (and  the  operational  concepts  which  will  derive  from  its  adoption) 
which  must  be  integrated  into  other  NATO  T&E  programs.  There  is  currently  no  NATO  body  which  has  the 
mission  to  support  or  to  carry  out  this  mission. 

After  a  careful  inspection  of  all  medically-related  organizations  within  the  NATO  Alliance,  we  have  come  to 
the  conclusion  that  the  only  suitable  organization  to  be  tasked  with  such  a  mission  is  that  of  the  medical 
branch  at  Allied  Command  Transformation  (ACT).  Since  the  real  question  involved  in  such  T&E  is  how  the 
introduction  of  the  new  medical  equipment  will  fit  into  or  affect  NATO  doctrine,  and  since  ACT  is  the 
organization  directly  responsible  for  the  development  and  adoption  of  new  doctrine,  it  appears  evident  that 
testing  the  suitability  of  such  new  equipment  and  its  effects  on  NATO  doctrine  most  appropriately  falls  to 
ACT.  ACO  is  tasked  to  implement  the  doctrine,  and  has  input  into  the  development  of  new  doctrine  based  on 
operational  experience  and  lessons  learned,  but  it  remains  ACT’s  responsibility  to  actually  develop  the  new 
concepts  and  doctrine.  NAMSA  may  undertake  the  acceptance  testing  of  any  medical  equipment  which 
NATO  decides  to  purchase  in  the  future  to  ensure  compliance  with  procurement  contracts  and  functionality, 
but  they  do  not  have  the  mission  or  the  capability  of  carrying  out  or  planning  the  type  of  operational  T&E 
which  we  feel  is  required.  We  recognize  that  this  new  mission  assignment  might  demand  increased  personnel 
and  budget  allocations  for  ACT.  It  must  be  noted  that  ACT  Medical  Section  does  not  agree  with  this 
concept  of  responsibility  assignment,  though  they  have  no  viable  alternatives  to  propose.  They  are  of  the 
opinion  that  they  currently  have  no  tasking  for  this  mission,  nor  do  they  have  adequate  personnel  and 
budget  to  carry  it  out.  We  are  in  complete  agreement,  and  agree  that  in  order  to  carry  out  our 
recommendation,  ACT  Medical  Branch  would  need  to  receive  a  formal  tasking  for  this  mission, 
and  would  need  to  have  increased  personnel  and  budget  authorized  for  this  purpose.  That  need, 
however,  does  not  invalidate  our  recommendation  as  to  our  belief  that  ACT  is  the  most  appropriate 
body  to  be  given  this  mission. 

The  previous  recommendation  does  not,  of  course,  mean  that  ACT  personnel  must  have  the  direct 
responsibility  for  carrying  out  such  tests  -  it  simply  means  that  any  such  testing  should  be  carried  out  at  the 
direction  and  under  the  general  supervision  of  ACT.  Such  evaluation  should  be  carried  out  under  the  authority 
of  ACT,  and  should  be  included  in  its  Experimentation  Programme  of  Work  (EPOW)  or  its  exercise  program. 
The  actual  carrying  out  of  the  evaluation  could  be  tasked  to  other  agencies  as  appropriate.  ACT  already  has  its 
work  in  materiel  development  (e.g.,  the  MEDICS  project)  contracted  out  to  NC3A,  and  in  the  past  the  Medical 
Communications  and  Information  Systems  (MedCIS)  Expert  Panel  has  carried  out  development  of  business 
process  models  for  various  systems  on  behalf  of  ACT,  as  part  of  the  exercise  program.  In  the  future,  it  is 
possible  that  the  proposed  Medical  Center  of  Excellence  (Med  COE)  could  undertake  some  aspects  of  this 
work  on  behalf  of  ACT.  In  the  realization  that  NATO  may  decide  not  to  assign  this  responsibility  to  ACT  in 
the  future,  throughout  this  document  we  have  referred  to  the  organization  responsible  for  testing  new  medical 
equipment  as  the  “NATO  Testing  Organization”  (NTO),  to  avoid  confusion.  In  spite  of  that,  we  do  believe 
that  the  ACT  is  the  most  appropriate  organization  to  be  tasked  with  the  responsibility  of  serving  as  the  NTO. 

2.2  HEALTH  TECHNOLOGY  ASSESSMENT  (HTA) 

During  the  first  meeting  of  this  RTG,  an  unspoken  assumption  was  that  our  task  was  to  develop  a  Health 
Technology  Assessment  (HTA)  program  for  NATO  along  the  lines  of  those  currently  in  use  in  several  nations 
and  internationally4.  However,  on  careful  analysis  of  our  task,  and  after  examining  the  HTA  literature, 

4  The  Reference  List  contains  many  references  referring  to  this  concept. 
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we  realized  that  our  task  was  NOT  to  develop  a  NATO  Health  Technology  Assessment  program  -  HTA  as 
normally  carried  out  involves  both  economic  and  clinical  benefit  analysis  to  assist  a  health  service  in  making  a 
procurement  decision  -  all  we  are  looking  at  within  NATO  are  operational  benefit  and  operational  usability. 
While  there  are  certainly  aspects  of  the  HTA  process  which  are  relevant  to  this  effort  (as  detailed  in  the 
reference  list),  HTA  is  fundamentally  a  research-based,  applied  assessment  of  all  relevant  available 
knowledge  of  clinical  problems  and  their  potential  solutions,  including  overall  clinical  benefit  and  cost-benefit 
analysis.  It  is  not  only  an  assessment  of  clinical  or  operational  capabilities,  but  is  also  directed  at  policy¬ 
making,  and  has  strong  relationships  with  planning,  administration,  budgeting,  and  management  due  to  a 
focus  on  decision-making.  Thus,  HTA  not  only  addresses  the  issue  of  “does  this  technology  do  what  it  is 
supposed  to  do,  and  is  it  both  beneficial  and  usable?”,  but  also  the  issue  of  “should  our  organization  adopt  this 
technology  from  a  policy/budget  standpoint?”  The  HTA  also  takes  into  account  results  obtained  from 
randomized  controlled  clinical  and/or  epidemiological  trials,  examining  such  elements  as  competing  clinical 
technologies,  side  effects,  and  cost-benefit  analyses.  While  of  course  of  interest  to  those  nations  considering 
adoption  of  these  technologies  for  use  within  their  own  national  health  systems  or  military  medical  systems, 
many  of  the  issues  involved  in  a  true  HTA  are  not  really  relevant  to  the  NATO  environment,  as  one  or  more 
of  the  nations  will  have  already  decided  to  purchase  and  use  this  technology  before  offering  it  for  use  in  the 
NATO  multi-national  medical  environment.  Thus,  our  approach  has  been  to  look  at  this  issue  more  along  the 
lines  of  operational/usability  testing  than  those  of  a  full  HTA.  NATO  simply  has  no  current  capability  for 
carrying  out  full  HTA  programs  (which  historically  can  take  9-36  months  per  analysis5),  and  in  doing  so 
would  be  intruding  on  national  prerogatives  to  manage  their  own  military  health  care  systems.  Thus,  we  have 
chosen  to  not  attempt  to  develop  an  HTA  mechanism  for  use  within  the  Alliance. 

On  the  other  hand,  many  of  the  elements  and  functions  of  an  HTA  have  direct  applicability  to  the  NATO 
scenario6 7 8'7,8.  The  concepts  of  question  development  prior  to  study,  and  various  testing  programs,  have  direct 
cross-over  utility  for  NATO,  even  when  the  final  outcome  is  desired  to  be  less  complex  than  a  full  HTA. 


2.3  TECHNOLOGY  READINESS  ASSESSMENT  (TRA) 

A  Technology  Readiness  Assessment  (TRA)  is  one  sub-set  of  the  full  group  of  analyses  included  in  a  HTA, 
but  is  one  which  also  has  utility  outside  of  the  HTA  concept.  A  TRA  can  be  defined  as  a  systematic  and 
metrics-based  process  which  is  used  to  assess  the  maturity  of  certain  technologies  before  moving  on  to  further 
development  or  fielding.  It  can  be  applied  to  either  devices  or  to  Critical  Technology  Elements  in  a  system. 
In  our  context,  it  can  be  considered  a  pre-screening  of  new  medical  equipment  to  determine  whether  or  not 
that  equipment  is  technologically  advanced  enough  to  enter  the  below-proposed  NATO  medical  equipment 
Test  and  Evaluation  system. 

Many  nations  currently  use  a  system  called  evolutionary  acquisition  or  “spiral  development”,  which  enables 
the  rapid  fielding  of  an  initial  capability  to  the  end-user,  followed  by  new  versions  with  incremental 
improvements  in  capability.  In  the  past,  many  requirements  developers  established  extremely  detailed  and 
challenging  performance  requirements  which  often  resulted  in  long,  high-risk,  and  expensive  development 
and  acquisition  programs.  Evolutionary  acquisition  uses  more  realistic  requirements  which  will  enable  the 


For  example,  Danish  Centre  for  Evaluation  and  Health  Technology  Assessment  (DACEHTA);  Introduction  to  Mini-HTA. 

6  For  example,  Canadian  Agency  for  Drugs  and  Technologies  in  Health.  Guidelines  for  the  Economic  Evaluation  of  Health 
Technologies. 

7  For  example,  Goodman,  C.,  et  al.  Health  Care  Technology  Assessment  in  VA. 

8  Kristensen,  F.B.,  Horder,  M.  and  Poulsen,  P.B.  (Eds.),  Health  Technology  Assessment  Handbook. 
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rapid  fielding  of  an  initial  capability  to  the  end-user,  followed  by  new  versions  with  incremental 
improvements  in  capability.  At  the  same  time  that  the  equipment  is  being  developed  and  fielded, 
the  government  communities  for  requirements,  acquisition,  R&D,  and  sustainment  work  together  as  a  team 
along  with  the  technology  developer,  to  refine  the  details  of  the  system  and  agree  on  tradeoffs  needed  to  make 
the  system  both  more  optimally  effective  and  affordable.  While  a  system  is  being  developed,  the  requirements 
producers  identify  the  essential  capabilities  needed  but  allow  the  technology  developers  the  flexibility  to 
determine  how  the  need  is  met.  Giving  all  involved  the  largest  possible  flexibility  enables  innovation  and  the 
balancing  of  performance,  operational  and  support  characteristics.  Thus,  a  technology  which  provides  an  80% 
solution  may  be  fielded  and  used  pending  development  of  the  100%  solution,  rather  than  waiting  for  the  100% 
solution  to  become  technologically  viable. 

This  system  of  development  is  extremely  useful  and  valid  when  the  end-user  and  the  technological  developers 
always  work  within  the  same  system,  but  it  poses  significant  potential  threats  to  a  system  such  as  NATO. 
If  the  end-user  is  not  from  the  same  country  as  the  developer,  and  if  the  requirements  generator  has  not  taken 
into  account  Alliance  (as  versus  strictly  national)  doctrine,  then  it  is  possible  that  the  technology  or  system 
which  works  quite  well  in  the  national  system  of  “spiral  development”  may  not  be  understood  or  work  as  well 
in  the  context  of  NATO  doctrine. 

One  central  theme  of  the  materiel  development  process  is  that  the  technology  employed  in  system 
development  should  be  adequately  “mature”  before  system  development  begins.  Normally,  for  technology  to 
be  considered  mature,  it  must  have  been  applied  in  a  prototype  article  (a  system,  sub-system,  or  component), 
tested  in  a  relevant  or  operational  environment,  and  found  to  have  performed  adequately  for  the  intended 
application.  This  implies  a  need  for  a  way  to  measure  maturity  and  for  a  process  to  ensure  that  only 
sufficiently  mature  technology  is  employed9.  In  the  absence  of  such  a  mechanism,  it  is  possible  to  waste  a 
significant  amount  of  time  and  money  testing  equipment  or  devices  which  are  later  found  to  not  be  sufficiently 
technologically  advanced  for  production  and  deployment.  A  TRA  is  a  systematic,  metrics-based  process 
which  assesses  the  maturity  of  critical  technology  elements.  The  TRA  should  be  conducted  before  entering  a 
technology  into  a  NATO  T&E  Program.  If  a  platform  or  system  depends  on  specific  technologies  to  meet 
system  operational  threshold  requirements  in  development,  production,  and  operation,  and  if  the  technology  or 
its  application  is  either  new  or  novel,  then  that  technology  is  considered  a  Critical  Technology  Element. 
The  TRA  is  not  a  risk  assessment  per  se  as  detailed  in  Chapter  6,  but  it  should  be  viewed  as  a  tool  for 
assessing  the  adequacy  of  technological  maturity.  The  TRA  scores  the  current  readiness  level  of  a  selected 
technology  or  system,  using  defined  Technology  Readiness  Levels  (see  below,  and  Annex  A).  The  TRA 
essentially  “draws  a  line  in  the  sand”  on  the  day  of  the  evaluation  to  allow  making  an  assessment  of 
technology  readiness  for  critical  technologies. 

It  is  through  a  Technology  Readiness  Assessment  that  the  Alliance  can  determine  whether  a  technology  is 
sufficiently  advanced  to  allow  its  effective  evaluation  for  use  in  the  NATO  multi-national  medical  environment. 
If  the  determination  of  a  TRA  were  that  the  system  does  not  meet  pre-defined  Technology  Readiness  Level 
scores,  then  further  testing  in  the  NATO  T&E  program  would  not  be  indicated  at  that  time. 


2.4  TECHNOLOGY  READINESS  LEVELS  (TRLs) 

The  concept  of  Technology  Readiness  Levels  is  simply  that  of  a  uni-dimensional  scale  used  to  provide  a 
measure  of  technological  maturity  at  a  given  point  in  time.  Its  use  can  provide  a  repeatable  system  for 
measuring  a  technology’s  maturity  by  identifying  a  “Snap  shot”  of  technological  maturity  at  the  time  of 
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evaluation.  The  use  of  TRLs  is  therefore  fundamental  to  any  evaluation  of  new  technology  in  our 
environment.  A  TRL  value  indicates  only  what  has  already  been  accomplished  in  the  development  of  a 
technology  -  it  does  not  indicate  that  the  technology  is  right  for  the  job  or  that  application  of  the  technology 
will  result  in  successful  development  of  the  system.  Neither  does  it  address  the  questions  of  “Can  it  be  used, 
and  does  it  do  what  it  is  expected  to?”  or  “How  does  it  affect  doctrine?”  -  Those  questions  must  be  answered 
by  an  operational  test  and  evaluation  system. 

Technology  Readiness  Levels  were  originally  developed  by  The  United  States’  National  Aeronautics  and 
Space  Administration  (NASA)  in  the  late  1980s  (Table  2-1).  The  original  definitions  only  included  seven 
levels,  which  were  clearly  defined  so  as  to  apply  to  the  NASA  mission.  These  seven  levels  were  later 
expanded  to  nine  levels,  as  experience  showed  the  deficiencies  in  the  use  of  only  the  original  seven  levels  in 
an  operational  setting. 


Table  2-1 :  Original  NASA  TRL  Definitions 


Though  NASA  successfully  used  these  TRLs,  other  agencies  which  looked  at  the  system  determined  that  for 
ease  of  use  and  uniformity  of  application,  the  definitions  characterizing  each  level  needed  to  be  more  detailed 
and  comprehensive.  The  United  States  Air  Force  adopted  the  use  of  Technology  Readiness  Levels  in  the 
1990s,  but  they  did  not  become  widely  used  in  the  USAF  until  1995,  when  John  C.  Mankins  of  NASA 
proposed  much  more  detailed  descriptions  for  each  TRL11.  In  1999,  the  United  States  General  Accounting 
Office  (GAO)  produced  an  influential  report12  which  examined  the  differences  in  technology  transition 
between  the  United  States  DoD  and  private  industry.  It  concluded  that  the  DoD  took  greater  risks  and 
attempted  to  utilize  emerging  technologies  at  lesser  degrees  of  maturity  than  did  private  industry.  The  GAO 
concluded  that  use  of  immature  technology  increased  overall  program  risk,  and  recommended  that  the  entire 
DoD  adopt  the  use  of  NASA’s  TRLs  as  a  means  of  assessing  technology  maturity  prior  to  transition.  Finally, 
in  2001,  the  Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  issued  a  memorandum  which 
endorsed  the  use  of  TRLs  in  all  new  major  programs,  which  was  subsequently  incoiporated  into  DoD 
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acquisition  Guidance13,14.  The  following  TRLs  are  now  in  use  in  all  non-medical  acquisition  and  materiel 
development  programs  within  the  United  States  Department  of  Defense  (DoD)  (Table  2-2). 

Table  2-2:  U.S.  DoD-Defined  TRLs15 


Level  Description/Supporting  Information 


1)  Basic  principles  Lowest  level  of  technology  readiness, 

observed  and  reported  Scientific  research  begins  to  be  translated  into  applied  research  and 

development.  Examples  might  include  paper  studies  of  a  technology’s 
basic  properties. 

Published  research  that  identifies  the  principles  that  underlie  this  technology. 
References  to  who,  where,  when. 


2)  Technology  concept 
and/or  application 
formulated 


3)  Analytical  and 
experimental  critical 
function  and/or 
characteristic  proof  of 
concept 


4)  Component  and/or 
breadboard  validation  in 
[a]  laboratory 
environment 


Invention  begins.  Once  basic  principles  are  observed,  practical  applications 
can  be  invented. 

Applications  are  speculative,  and  there  may  be  no  proof  or  detailed  analysis 
to  support  the  assumptions. 

Examples  are  limited  to  analytic  studies. 

Publications  or  other  references  that  outline  the  application  being  considered 
and  that  provide  analysis  to  support  the  concept. 

Active  research  and  development  is  initiated.  This  includes  analytical  studies 
and  laboratory  studies  to  physically  validate  analytical  predictions 
of  separate  elements  of  the  technology. 

Examples  include  components  that  are  not  yet  integrated  or  representative. 

Results  of  laboratory  tests  performed  to  measure  parameters  of  interest  and 
comparison  to  analytical  predictions  for  critical  sub-systems. 

References  to  who,  where,  and  when  these  tests  and  comparisons  were 
performed. 

Basic  technological  components  are  integrated  to  establish  that  they  will  work 
together.  This  is  relatively  “low  fidelity”  compared  to  the  eventual  system. 

Examples  include  integration  of  “ad  hoc”  hardware  in  the  laboratory. 

System  concepts  that  have  been  considered  and  results  from  testing 
laboratory-scale  breadboard(s). 


References  to  who  did  this  work  and  when. 


Provide  an  estimate  of  how  breadboard  hardware  and  test  results  differ 
from  the  expected  system  goals. 


Defense  Acquisition  University,  Interim  Defense  Acquisition  Guidebook. 

Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  (DUSD  (S&T)),  Technology  Readiness  Assessment  (TRA) 
Deskbook. 

Defense  Acquisition  University,  Interim  Defense  Acquisition  Guidebook. 
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Level 

5)  Component  and/or 
breadboard  validation  in 
[a]  relevant  environment 


6)  System/sub-system 
model  or  prototype 
demonstration  in  a 
relevant  environment 


7)  System  prototype 
demonstration  in  an 
operational  environment 


8)  Actual  system 

completed  and  qualified 
through  test  and 
demonstration 


Description/Supporting  Information 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic 
technological  components  are  integrated  with  reasonably  realistic  supporting 
elements  so  they  can  be  tested  in  a  simulated  environment. 

Examples  include  “high-fidelity”  laboratory  integration  of  components. 

Results  from  testing  a  laboratory  breadboard  system  that  are  integrated  with 
other  supporting  elements  in  a  simulated  operational  environment.  How  does 
the  “relevant  environment”  differ  from  the  expected  operational 
environment?  How  do  the  test  results  compare  with  expectations? 

What  problems,  if  any,  were  encountered?  Was  the  breadboard  system 
refined  to  match  the  expected  system  goals  more  nearly? 

Representative  model  or  prototype  system,  which  is  well  beyond  that  of 
TRL  5,  is  tested  in  a  relevant  environment. 

Represents  a  major  step  up  in  a  technology’s  demonstrated  readiness. 
Examples  include  testing  a  prototype  in  a  high  fidelity  laboratory 
environment  or  in  [a]  simulated  operational  environment. 

Results  from  laboratory  testing  of  a  prototype  system  that  is  near  the  desired 
configuration  in  terms  of  performance,  weight,  and  volume. 

How  did  the  test  environment  differ  from  the  operational  environment? 

Who  performed  the  tests?  How  did  the  test  compare  with  expectations? 

What  problems,  if  any,  were  encountered?  What  are/were  the  plans, 
options,  or  actions  to  resolve  problems  before  moving  to  the  next  level? 

Prototype  near,  or  at,  planned  operational  system. 

Represents  a  major  step  up  from  TRL  6,  requiring  demonstration  of  an  actual 
system  prototype  in  an  operational  environment  such  as  an  aircraft,  vehicle, 
or  space. 

Examples  include  testing  the  prototype  in  a  test  bed  aircraft. 

Results  from  testing  a  prototype  system  in  an  operational  environment. 

Who  performed  the  tests?  How  did  the  test  compare  with  expectations? 

What  problems,  if  any,  were  encountered?  What  are/were  the  plans, 
options,  or  actions  to  resolve  problems  before  moving  to  the  next  level? 

Technology  has  been  proven  to  work  in  its  final  form  and  under  expected 
conditions. 

In  almost  all  cases,  this  TRL  represents  the  end  of  true  system  development. 

Examples  include  developmental  test  and  evaluation  of  the  system  in  its 
intended  weapon  system  to  determine  if  it  meets  design  specifications. 

Results  of  testing  the  system  in  its  final  configuration  under  the  expected 
range  of  environmental  conditions  in  which  it  will  be  expected  to  operate. 

Assessment  of  whether  it  will  meet  its  operational  requirements. 
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Since  acceptance  by  the  U.S.  DoD,  the  concept  of  TRLs  has  been  accepted  by  many  other  program  managers, 
agencies,  corporations,  and  nations.  The  concept  of  the  use  of  TRLs  as  a  tool  to  describe  the  maturity  of 
technologies  being  considered  for  new  systems  or  for  fielding  is  now  widely  accepted  in  materiel  development 
and  acquisitions  programs  worldwide.  NATO  has  never  formally  accepted  the  concept  of  TRLs,  though  the 
NATO  Undersea  Research  Center  (NURC)  has  developed  a  set  of  TRLs  which  they  have  recommended  be 
adopted  by  the  Allied  Command  Transformation’s  (ACT)  Technology  Advisory  Board  (TAB)16.  These 
proposed  TRL  definitions  go  beyond  those  currently  adopted  by  the  U.S.  DoD,  and  are  felt  by  the  NURC  to 
be  more  compatible  with  NATO  needs.  Unlike  the  NASA  originals,  each  readiness  level  is  defined  in  general 
terms  according  to  the  state  of  technological  development,  and  is  additionally  described  in  detail  to  assist  the 
user  in  determining  into  which  category  a  given  technology  may  fall  (Table  2-3).  (It  has  been  reported  to  us 
that  these  may  have  already  been  approved  in  concept  by  RTO,  but  we  can  find  no  evidence  of  this  in  RTO 
documentation.  In  any  case,  they  are  not  quite  suitable  for  biomedical  usage  as  presented.) 


Table  2-3:  NURC-Recommended  NATO  TRLs17 


16  NURC,  NATO  Technology  Readiness  Levels. 

17  Ibid. 
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NATO  Technology  Description 

Readiness  Level 


3)  Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of  concept 


Analytical  studies  and  laboratory/field  studies  to  physically  validate 
analytical  predictions  of  separate  elements  of  the  technology  are 
undertaken.  Example  R&T  outputs  include  software  or  hardware 
components  that  are  not  yet  integrated  or  representative  of  final 
capability  or  system. 


4)  Component  and/or  breadboard  Basic  technology  components  are  integrated.  This  is  relatively  low 
validation  in  laboratory/field  fidelity  compared  to  the  eventual  system.  Examples  of  R&T  results 

(e.g.,  ocean)  environment  include  integration  and  testing  of  ad  hoc  hardware  in  a  laboratory/ 

field  setting.  Often  the  last  stage  for  R&T  (funded)  activity. 


5)  Component  and/or  breadboard  Fidelity  of  sub-system  representation  increases  significantly, 
validation  in  a  relevant  The  basic  technological  components  are  integrated  with  realistic 

(operating)  environment  supporting  elements  so  that  the  technology  can  be  tested  in  a  simulated 

operational  environment. 


Examples  include  high  fidelity  laboratory/field  integration  of 
components.  Rarely  an  R&T  (funded)  activity  if  it  is  a  hardware  system 
of  any  magnitude  or  system  complexity. 


6)  System/sub-system  model  or 
prototype  demonstration  in  a 
realistic  (operating) 
environment  or  context 


Representative  model  or  prototype  system,  which  is  well  beyond  the 
representation  tested  for  TRL  5,  is  tested  in  a  more  realistic  operational 
environment.  Represents  a  major  step  up  in  a  technology’s 
demonstrated  readiness.  Examples  include  testing  a  prototype  in  a  high 
fidelity  laboratory/field  environment  or  in  simulated  operational 
environment.  Rarely  an  R&T  (funded)  activity  if  it  is  a  hardware  system 
of  any  magnitude  or  of  significant  system  complexity. 


7)  System  prototype 
demonstration  in  an 
operational  environment  or 
context  (e.g.,  exercise) 


Prototype  near  or  at  planned  operational  system  level.  Represents  a 
major  step  up  from  TRL  6,  requiring  the  demonstration  of  an  actual 
system  prototype  in  an  operational  environment,  such  as  in  a  relevant 
platform  or  in  a  system-of-systems.  Information  to  allow  supportability 
assessments  is  obtained.  Examples  include  extensive  testing  of  a 
prototype  in  a  test  bed  vehicle  or  use  in  a  military  exercise.  Not  R&T 
funded  although  R&T  experts  may  well  be  involved. 


8)  Actual  system  completed  and 
qualified  through  test  and 
demonstration 


Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents  the  end 
of  Demonstration.  Examples  include  test  and  evaluation  of  the 
system  in  its  intended  weapon  system  to  determine  if  it  meets  design 
specifications,  including  those  relating  to  supportability.  Not  R&T 
funded  although  R&T  experts  may  well  be  involved. 


9)  Actual  system  operationally  Application  of  the  technology  in  its  final  form  and  under  mission 

proven  through  successful  conditions,  such  as  those  encountered  in  operational  test  and  evaluation 

mission  operations  and  reliability  trials.  Examples  include  using  the  final  system  under 

operational  mission  conditions. 


©  2004  -  All  rights  reserved  -  NATO  Undersea  Research  Centre. 
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Normally,  the  definitions  of  and  references  to  TRLs  are  almost  always  in  engineering  terms,  but  the  TRL 
concept  is  applicable  to  any  technology.  Medically-related  items  require  TRL  definitions  and  descriptions  that 
are  appropriate  to  the  technologies  upon  which  they  are  based,  and  that  account  for  the  specialized  statutes 
and  regulations  which  govern  their  development  and  use.  Neither  the  NASA,  the  DoD,  nor  the  NURC  TRL 
definitions  are  directly  suitable  for  use  in  evaluating  the  status  of  biomedical  technologies.  In  many  ways, 
biomedical  technologies  are  more  complex  than  the  vast  majority  of  engineering  and  design  technologies 
involved  in  even  space  flight.  There  are  several  categories  of  biomedical  technologies  which  each  demand 
modified  TRLs  to  appropriately  assess  them.  Further,  most  biomedical  technologies  are  subject  to  stringent 
regulatory  controls  (e.g.,  the  U.S.  Food  and  Drug  Administration  (FDA)  or  the  European  Union’s  European 
Medicines  Agency  (EMEA)),  which  must  be  applied  at  various  levels  of  development.  Thus,  taking  these 
regulatory  requirements  into  account  is  also  a  requirement  when  assessing  whether  a  biomedical  product 
(whether  drugs,  vaccines,  or  equipment)  is  ready  for  NATO  use.  In  searching  for  a  TRL  set  which  is  suitable 
for  use  in  the  biomedical  arena,  the  most  useful  one  we  have  found  is  that  developed  for  the  U.S.  Army 
Medical  Research  and  Materiel  Command18.  However,  this  particular  set  of  TRLs  seems  too  U.S. -centric  for 
NATO  use  as  originally  prepared.  Accordingly,  the  RTG  has  modified  this  set  of  biomedical  TRLs  for  NATO 
use  by  referring  to  generic  national  or  EU  approval  bodies  and  procedures  rather  than  the  U.S.  FDA  and 
U.S. -specific  development  processes,  etc.  There  are  five  categories  of  biomedical  products,  which  each  have 
differing  technological  and  developmental  requirements.  Thus,  the  TRLs  for  each  are  a  bit  different.  In  much 
medical  technology,  software  is  considered  an  integral  part  of  the  system  or  sub-system  in  which  it  operates. 
Therefore,  demonstration  of  a  technology  at  the  sub-system  or  system  level  must  include  demonstration  of  the 
associated  software.  Although  at  the  present  time  we  believe  that  the  potential  for  NATO  as  an  alliance 
becoming  involved  in  drug  or  vaccine  development  is  remote,  we  cannot  eliminate  it  as  a  future  possibility. 
Therefore,  although  the  primary  purpose  of  our  proposed  T&E  system  is  to  analyze  the  other  three  categories 
of  biomedical  developments,  we  have  also  provided  TRLs  for  both  drug  and  vaccine  development  in  the 
interests  of  completeness,  either  for  Alliance  use  or  for  use  by  nations  which  wish  to  make  use  of  this  concept. 
Additionally,  we  have  provided  clear  descriptions  of  those  evaluation  criteria  which  can  be  used  to  define  at 
which  TRL  a  technology  is  currently  found.  It  should  be  noted  that  activities  described  as  occurring  between 
successive  TRL  Decision  Criteria  are  intended  to  exemplify  the  kinds  of  activities  that  routinely  take  place 
when  maturation  is  sequential  and  stepwise  -  these  examples  are  neither  mandatory  nor  all  inclusive. 
The  lower  a  critical  technology’s  TRL  when  transitioning  from  technology  development  to  product 
development,  the  greater  the  risks  of  failure.  For  medical  technologies,  risk  reduction  is  not  linear  across 
TRLs.  The  rate  of  risk  reduction  remains  very  low  until  very  late  in  development.  We  propose  that  the 
biomedical  TRL  matrix  shown  in  Annex  A  be  formally  adopted  by  NATO,  and  used  as  the  basis  for  screening 
of  biomedical  technology  before  entry  into  the  proposed  NATO  T&E  system  proposed  below.  Further, 
we  recommend  that  no  item  or  technology  be  accepted  for  testing  in  the  below-recommended  T&E  Scheme 
unless  it  is  adjudged  to  be  at  least  at  TRL  6. 


Sciences  Applications  International  Corporation.  "Biomedical  Technology  Reference  Levels  (TRLs)”. 
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There  are  many  different  aspects  of  medical  equipment  testing,  each  with  their  own  techniques  and 
requirements.  In  deciding  to  undertake  a  medical  Test  and  Evaluation  program,  decisions  must  be  made  as  to 
what  type  of  testing  is  desired  or  needed  prior  to  developing  the  test  plan. 

3.1  FUNCTIONAL  TESTING 

3.1.1  Functional  Tests 

Functional  tests  focus  mainly  on  matching  expected  system  behavior  against  an  agreed  functional 
specification  (inputs,  system  behavior  and  expected  outputs).  Experimental  tests  or  observational  study 
outcomes  can  be  derived  by  box-ticking  lists  of  a  set  of  criteria,  user  group  interviewing  or  observation  of  end 
users  using  the  system  and  logging  of  system  responses.  The  test  specification  (criteria,  measured  variables, 
use  cases  and  testing  methodology)  must  be  agreed  during  the  system  definition  phase  and/or  negotiated  with 
the  client  as  project  deliverables  (success  criteria).  Chapter  4  should  be  consulted  for  details  of  developing 
such  specifications. 

3.1.2  Hardware  and  Software  Testing 

Hardware  and  software  testing  involve  different  applications  under  the  same  testing  rationale.  For  example  for 
stress  testing  of  software  components,  multiple  protocols  are  applied  under  heavy  load  and  usage  patterns  to 
observe  system  behavior  and  results,  while  hardware  testing  may  include  physical  parameters  such  as  high 
voltage,  frequencies,  processor  loads,  over-  and  under-clocking,  heat,  cold,  dampness,  etc.  A  large  number 
of  regulations  and  procedures  exists  for  domain-specific  tests  (e.g.,  FCC,  TUV,  CE,  EU  directives),  and  these 
policies  should  be  consulted  prior  to  developing  a  test  process. 

3.1.3  Non-Functional  Testing  of  Operation  of  the  Equipment 

3. 1.3.1  Performance  Testing 

Performance  testing  includes  an  assessment  and  analysis  of  response  time,  listed  output  measures, 
and  scalability.  Typically,  an  experimental  set-up  is  used  for  the  assessment  of  quantitative  measures  (these  are 
defined  during  the  system  development  phase,  particularly  in  a  test  plan  document). 

3. 1.3.2  Reliability  Testing 

Reliability  testing  is  performed  with  a  test  set-up  with  varying  expected  usage  patterns,  which  is  used  for  the 
collection  of  performance,  stability  and  effectiveness  data.  The  focus  can  be  on  detection  of  faults  for  later 
removal  (defect  testing)  or  the  focus  can  be  on  reliability  evaluation  for  gaining  confidence  in  system 
performance. 

3. 1.3.3  Stress  Testing 

Stress  testing  can  be  carried  out  by  means  of  an  experimental  study  where  system  behavior  is  assessed  by 
going  beyond  normal  operational  capacity  or  to  a  breaking  point.  Prospective  aspects  of  studies  involve 
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automated  script  based  testing  (e.g.,  bum-in  test)  and  by  tests  with  several  users  trying  out  various  usage 
scenarios  with  the  intent  of  faulting  the  system.  With  a  stable  delivered  system,  a  retrospective  analysis  of  logs 
of  real  usage  data  can  be  used. 

3. 1.3.4  Workload  Testing 

Workload  testing  involves  analysis  of  system  behavior  under  normal  operation  using  retrospective  analysis  of 
logs  of  real  usage  data.  Also,  automated  bum-in  testing  can  be  deployed. 

3. 1.3.5  Data  Integrity  Testing 

Data  integrity  testing  involves  the  experimental  testing  of  data  integrity  over  time  as  testing  under  various 
system  conditions  is  conducted.  Measures  of  completeness  and  accuracy  of  the  stored  data,  data  in  transit  and 
during  processing  are  collected  and  analyzed. 

3. 1.3.6  Error  Handling  Testing 

Error  handling  testing  evaluates  operator  or  user  performed  rules  or  use  case  based  testing  (domain  specific 
standards  or  mles  predefined  during  development  and  resulting  mles-scenarios  from  stress  testing)  of  system 
performance  in  internal  and  external  error  events.  Data  covering  recovery  behaviors,  information  about  status 
and  user  guidance  variables  are  collected  and  analyzed. 

3. 1.3.7  Recovery  Testing 

Recovery  testing  involves  experimental,  operator-led  study  of  system  ability  to  recover  from  crashes, 
hardware  failures,  etc.  In  addition  log-based  retrospective,  observational  studies  are  conducted  after  system 
delivery. 


3. 1.3.8  Software  Regression  Testing 

Software  regression  testing  involves  tests  of  system  functioning  and  reliability  after  bug  fixing,  upgrading, 
parameter  changes,  etc.  Either  focused  tests  of  selected  affected  features  (use  cases)  or  full-scale  post-tests  for 
reliability,  performance  and  stress  are  performed. 

3. 1.3.9  Compatibility  Testing 

Compatibility  testing  requires  the  testing  of  standards  compliance,  interoperability,  connectivity,  interfacing, 
attached  or  co-used  hardware,  backwards  compatibility,  etc.  -  i.e.,  compatibility  within  the  working/functional 
environment. 

3.1.3.10  Maintenance  Testing 

Maintenance  testing  involves  the  analysis  of  typical  use-related  problem  identification  (system  responses, 
feedback,  status  reports),  cost  and  performance  data  for  preventive  and  corrective  maintenance,  post-repair 
reliability  control,  and  quality  control. 

3.1.3.11  Security  Testing 

Security  testing  utilizes  experimental  tests  assessing  the  system’s  ability  to  contain  data  integrity, 
data  authentication,  user  authorization,  confidentiality,  later  non-repudiation  of  user  or  illicit  actions  and 
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system’s  ability  to  uphold  functionality.  Tests  involve  software  based  and  physical  break-ins,  hacking,  known 
software  bug  exploitations. 

3.1.3.12  Usability  Testing 

Usability  testing  involves  various  tests  measuring  the  system’s  capacity  to  meet  its  intended  purpose, 
highlighting  ease  of  use,  efficiency  of  usage,  accuracy  of  usage,  ease  of  learning  and  recalling,  satisfying 
pleasing  and  emotional  values  of  end  users,  etc.  The  goals  of  usability  testing  fall  either  into  the  creation  of 
records  of  usability  benchmarks  for  future  releases  or  for  minimizing  the  cost  of  service  and  support; 
minimizing  risk  of  adverse  behaviour,  etc.  Tested  and  monitored  variables  are  selected  according  to  above 
goals.  Furthermore,  usability  testing  can  be  seen  of  having  following  distinctions: 

•  Exploratory  tests  engage  focus  groups  to  evaluate  the  effectiveness  of  design  concepts.  Observer 
lead  user  querying  and  monitoring  tests  are  performed. 

•  Assessment  tests  evaluate  usability  of  lower-level  operations  and  aspects  the  system.  Quantitative 
measures  of  screens,  buttons,  or  clicks  performed  without  an  observer’s  intervention  are  collected  and 
analyzed. 

•  Validation  tests  compare  usability  against  predetermined  usability  benchmarks  (internal  company  or 
external  standards.  Heuristics  checklist,  quantitative  data  from  user  behavior  performing  given  tasks, 
focus  on  data  collection,  but  also  situations  of  sub-standard  performance  are  identified. 

•  Comparison  tests  are  used  in  conjunction  with  other  usability  tests  to  compare  two  or  more 
alternative  designs  with  current  system.  Alternatives  can  be  prototypes  or  existing  products. 

•  Accessibility  testing  are  tests  assessing  usability  criteria  in  varying  environmental  conditions  (light  and 
dark  lighting  conditions,  in-  and  out-doors,  noisy  environment,  during  movement,  etc.),  tests  assessing 
system  usability  for  people  with  special  needs,  etc. 

•  User  help  testing  is  an  experimental  study  of  user  support  functionality  -  an  observational  study  of 
users  needing  help,  their  actions  accessing,  using  and  understanding  support  system,  and  results  in  a 
user  satisfaction  assessment. 

•  Installation  testing  is  the  review  of  tasks  of  system  set-up  by  the  user  in  the  user’s  environment. 
It  provides  measures  regarding  reliability,  ease  of  use,  resource  and  knowledge  requirements,  etc., 
which  are  collected  and  analyzed. 

•  Documentation  testing  involves  judgment  of  the  effectiveness  and  usability  of  supporting 
documentation.  Document  comparison  against  others  of  the  same  type  is  one  possible  tool.  Examination 
of  typographical  errors  and  reconsideration  error  of  meaning  using  independent  readers,  focus  group 
testing  of  knowledge  after  reading  the  document,  and  focus  group  testing  of  finding  relevant 
information  are  other  modalities  used  in  this  context. 
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4.1  PURPOSE 

This  chapter  contains  information,  guidance,  and  best  practices  pertaining  to  systematic  test  and  evaluation 
(T&E)  and  related  subjects.  Our  goal  is  to  propose  an  operational  test  and  evaluation  system  to  be  used  in 
conjunction  with  future  NATO  medical  technologies  evaluation  which  can  be  implemented  by  NATO  bodies 
and  programs  as  recommended  above  in  Chapter  2.  We  are  not  discussing  routine  procurement  acceptance 
testing,  as  is  routinely  done  by  procurement  organizations,  though  for  completeness  that  is  discussed  briefly  in 
Annex  B.  This  test  system  is  designed  to  evaluate  mature  technologies  under  consideration  for  inclusion  in 
NATO  medical  systems  -  i.e.,  in  the  early  stages  of  the  acquisition  process.  Operational  testing  determines 
“if  things  (e.g.,  devices,  systems)  work  and  have  operational  benefits”.  It  is  not  the  intent  nor  is  it  within 
the  scope  of  this  document  to  recommend  or  to  build  a  new  T&E  structure.  Our  goal  is  to  provide 
guidance  and  support  to  those  who  may  have  to  carry  out  NATO  T&E  under  the  direction  of  ACT, 
or  whatever  other  NATO  agency  is  formally  tasked  to  carry  out  this  mission,  hereinafter  referred  to  as  the 
NATO  Test  Organization  (NTO). 


4.2  OTHER  MANDATORY  DOCUMENTS 

This  chapter  should  be  used  in  conjunction  with  policies  and  best  practices  from  other  NATO  groups  and 
international  test  organizations.  Advanced  Concept  Technical  Demonstrations  (ACTD)  which  show 
“how  things  work”,  rather  than  “if  things  work”  for  example,  share  many  test  principles  with  operational 
testing  and  so  NATO  demonstration  policy  and  doctrine  should  be  reviewed1,  along  with  Chapter  5  of  this 
report  (which  discusses  Medical  Technology  Demonstrations  (MTD)),  in  preparation  for  NATO  operational 
T&E.  Readers  might  also  wish  to  familiarize  themselves  with  the  NATO  and  member  nation  documents  in  the 
reference  list  associated  with  this  report. 


4.3  HOW  TO  USE  THIS  CHAPTER 

This  chapter  is  set  up  to  be  utilized  as  a  guide  regardless  of  whether  NATO  creates  a  specific  T&E  organization 
at  some  point  in  time,  assigns  these  responsibilities  to  an  existing  NATO  organization  [e.g.,  Allied  Command 
Transformation  (ACT)  or  the  NATO  Maintenance  and  Supply  Agency  (NAMSA)],  or  if  a  member  nation  test 
organization  performs  the  testing  on  behalf  of  the  Alliance.  At  the  current  time,  it  appears  as  though  NATO  is 
primarily  interested  in  field  demonstrations  so  operational  testing  of  this  nature  is  emphasized  in  this  technical 
report. 


4.4  THE  UNDENIABLE  NEED  FOR  T&E 

T&E  must  demonstrate  capabilities  today  that  will  be  needed  tomorrow  throughout  the  entire  spectrum  of 
NATO  missions  from  humanitarian  support  to  a  full  Article  5  operation.  When  a  capability  is  called  for  in  order 
to  provide  combat  medical  support,  the  need  is  immediate  and  there  is  no  time  to  consider  whether  or  not  the 
system  will  operate  as  designed  or  whether  it  can  be  effectively  used  by  all  potential  users  in  the  context  of  an 


1  For  example,  NATO  Guidance  for  Experimentation  Planning  (NAGEP)  Revision  2. 
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operational  scenario.  NATO  /  Partnership  for  Peace  (PfP)  military  personnel  and  their  leaders  need  assurance 
that  when  they  risk  their  lives,  the  medical  systems  provided  to  them  will  do  the  job  for  which  they  were 
designed. 

4.4.1  Desired  Outcomes 

The  desired  outcomes  of  T&E  programs  include:  product  maturity  evaluation  (discussed  more  fully  in  Chapter  2 
and  Annex  A),  management  of  risks  (Chapter  6),  identification  of  deficiencies  (Chapter  7),  and  assurance  that 
systems  are  operationally  effective  and  suitable  (Chapters  4  and  5).  The  NATO  T&E  community  should  conduct 
integrated  T&E  on  a  continuum  in  collaboration  with  ongoing  requirements  development  and  during  the  later 
phases  of  the  acquisition  process. 

4.4.2  Deficiency  Evaluation 

Credible  T&E  data  about  the  development  and  continued  sustainment  of  medical  systems  is  needed  in  order  to 
ensure  operational  readiness  in  the  multi-national  operational  environment.  If  equipment  deficiencies  are 
documented,  medical  planners  can  plan  around  these  deficiencies  and  medical  logisticians  can  attempt  to  remedy 
the  identified  deficiencies. 


4.5  PROPOSED  NATO  T&E  ORGANIZATIONAL  STRUCTURE 

4.5.1  Organization  in  Charge  of  Test  and  Evaluation 

At  the  time  of  this  publication,  the  authors  do  not  know  what  NATO  organization  will  be  placed  in  charge  of 
medical  T&E  programs.  Likely  candidates  for  this  task  are  the  Allied  Command  Transformation  (ACT)  or  the 
NATO  Maintenance  and  Supply  Agency  (NAMSA),  and  we  believe  that  ACT  would  be  the  most  appropriate 
agency  to  carry  out  this  task,  as  recommended  above.  For  the  purposes  of  this  document,  the  organization  in 
charge  of  medical  equipment  T&E  will  be  referred  to  as  the  NATO  Test  Organization  (NTO).  Note  that  in 
the  use  of  this  terminology  we  do  NOT  envision  the  creation  of  a  stand-alone  formal  T&E  organization, 
but  simply  recognizes  that  some  organization  must  be  formally  in  charge  of  any  such  program. 
The  tasks  identified  below  must  be  carried  out,  no  matter  which  organization  is  assigned  the  responsibility, 
and  failure  to  have  them  managed/supervised  by  a  single  organization  with  knowledge  and  responsibility  for 
the  functions  of  such  a  program  is  a  sure  route  to  lack  of  success. 

4.5.2  NATO  Test  Organization  (NTO)  T&E  Responsibilities 

The  NTO  should  be  assigned  the  following  responsibilities  related  to  the  fielding  of  medical  technologies: 
determine  the  need  for  T&E,  create  the  requirements  for  T&E,  create  the  test  plan,  assign/coordinate  for  some 
functional  body  to  carry  out  the  actual  T&E  process,  develop  the  assessment  metrics,  supervise  the  T&E, 
and  review  and  analyze  the  results  of  the  test  program.  Either  the  NTO  or  some  other  body  tasked  to  do  so 
must  undertake  an  oversight  role  of  the  process. 

4.5.3  Oversight  Functions 

Oversight  functions  include  requirements  that  the  NTO: 

•  Provide  early  involvement  in  drafting  the  T&E  strategy. 
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•  Supervise  testing  and  evaluate  the  impact  of  deploying  a  device  or  system  on  the  operational  mission. 
The  NATO  oversight  team  should  look  at  the  system  being  tested  with  all  interoperability  and 
supportability  considerations  taken  into  account;  then  ensure  that  systems  are  operationally  effective  and 
suitable  before  these  systems  go  to  fielding. 

•  Ensure  that  T&E  is  adequately  completed  prior  to  deployment  of  new  advanced  medical  technology, 
and  that  the  tests  were  properly  executed  according  to  NATO  guidance. 

4.5.4  NATO  T&E  Oversight 

NTO  criteria  to  be  considered  in  making  the  determination  as  to  whether  or  not  a  piece  of  equipment  or  a 
medical  system  should  be  tested  in  accordance  with  this  document  should  include: 

•  TRL  level  (see  Annex  A)  of  the  proposed  equipment  to  be  tested. 

•  Stage  of  development  or  production  of  the  item/device/system  being  evaluated. 

•  Whether  or  not  other  developmental  and/or  operational  testing,  the  results  of  which  can  be  utilized,  has 
been  done  in  Alliance  or  PfP  nations. 

•  Whether  or  not  there  is  inter-nation  compatibility. 

•  Whether  or  not  there  is  NATO/PfP  member  nation  interest  in  the  technology. 

•  Whether  or  not  the  technology  meets  regulatory  requirements  (refer  to  Annex  B). 

•  Relationship  with  other  medical  systems  as  part  of  a  system-of-systems. 

•  Technical  complexity  of  system  (e.g.,  Medical  Device  Classification,  ref.  Section  6. 1). 


4.6  T&E  SUPPORT  TO  THE  REQUIREMENTS  PROCESS 

4.6.1  Operational  Requirements  Development  and  Review 

In  general  terms,  when  NATO  is  considering  acquisition  or  fielding  of  new  medical  technology  on  an  Alliance 
basis,  Allied  Command  Operations  (ACO)  and/or  ACT  should  set  the  requirements,  while  the  NTO  or  another 
organization  tasked  by  the  NTO  should  do  the  testing.  Testing  organizations  utilized  for  NATO  testing  should 
be  involved  in  the  NATO  requirements  creation  process  if  possible.  A  suggested  NATO  requirements 
development  group  would  include  a  group  of  potential  users  and  subject  matter  experts  (SMEs).  Before 
developing  test  strategies  and  plans,  all  testers  should  thoroughly  understand  the  undeipinnings  of  their  test 
programs,  namely  the  operational  capability  requirements  from  which  that  testing  will  be  derived. 

4.6.2  Requirements  Development 

Testable  requirements  development  must  utilize  representatives  from  developmental  and  operational  test 
organizations  (those  who  will  actually  carry  out  the  testing)  as  well  as  doctrinal  experts  and  potential  end- 
users  (those  who  will  have  to  use  the  equipment)  in  order  to  achieve  optimal  representation.  The  test  planners 
will  determine  who  will  need  to  participate,  and  at  what  level  their  involvement  will  be  in  the  requirements 
development  process. 
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4.6.3  Review  of  Requirements  Policies  Impacting  T&E 

4.6.3. 1  Support 

The  requirements  strategy  must  reflect  the  required  capabilities  outlined  in  applicable  NATO  concepts  of 
operations,  NATO  policy  and  doctrine,  and  capability-based  planning  documents.  Applicable  medical  device 
classification  to  include  member  nation  regulations  should  be  considered  in  requirements  development. 
Regulatory  authorities  recognize  different  classes  of  medical  devices,  based  on  their  design  complexity,  their  use 
characteristics,  and  their  potential  for  harm  if  misused.  Each  country  or  region  defines  these  categories  in 
different  ways.  The  authorities  also  recognize  that  some  devices  are  provided  in  combination  with  drugs, 
and  regulation  of  these  combination  products  takes  this  factor  into  consideration  (ref.  Section  6.1).  Each  strategy 
must  be  tailored  to  address  strategy  elements  such  as:  member  nation  interoperability/implications,  funding,  test 
schedules,  testing  mechanisms,  supportability,  training,  analysis,  human  systems  integration,  potential  challenges 
and  constraints,  etc.  Following  the  development  of  the  requirements  strategy,  and  prior  to  documenting  the 
needed  capabilities  in  an  operational  requirements  document,  the  sponsor  presents  the  requirements  strategy  to 
the  NATO  leadership  for  approval. 

4.6.3.2  Requirements  Development 

The  NATO  leadership  should  assign  a  requirement  development  team  for  the  test  process  consisting  of 
Subject  Matter  Experts  (SMEs)  from  the  member  nations  or  from  other  appropriate  NATO  organizations.  It  is 
important  that  the  capabilities  described  in  the  test  requirement  documents  be  as  clear  and  detailed  as  possible. 
The  focus  is  on  understanding  the  required  capability  as  well  as  making  sure  the  requirements  are  testable. 
The  reader  should  refer  to  the  references  and  to  Annex  C  for  guidance  concerning  medical  technology 
requirements  development. 

4.6.4  The  Integrated  Testing  Process 

The  integrated  testing  process  is  a  well-proven  structured  system  for  identifying  testable  requirements, 
developing  a  concept  by  which  those  requirements  will  be  tested,  analyzing  the  results,  and  finally  reporting 
the  results  so  that  appropriate  actions  may  be  taken  by  NATO.  It  provides  a  road  map  which,  if  adequately 
developed  in  detail,  will  make  the  actual  carrying  out  of  the  test  or  project  much  easier  than  if  it  is  not 
followed. 
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Discovery 
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Test  Team  Formation 
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Test  Planning  Through  Results  Reporting  Stage 


NATO  Test  Approval 

JJ 

Test  Execution 

Test  Reporting 


Figure  4-1:  Outline  of  the  Integrated  T&E  Process. 


4.6.4.1  “Discovery” 

“Discovery”  is  the  process  by  which  NATO  leadership  becomes  aware  of  the  need  for  utilization  of  the  Test 
and  Evaluation  process.  This  information  may  arise  from  review  of  findings  by  the  NATO  Joint  Analysis  and 
Lessons  Learned  Centre  (JALLC),  from  decisions  of  the  COMEDS  Plenary,  from  the  Strategic  Commands, 
from  work  carried  out  by  the  Research  and  Technology  Organization  (RTO),  or  from  any  of  a  myriad  of  other 
NATO  Expert  Panels,  Expert  Teams,  and  Work  Groups. 

4.6.4.2  Test  Team  Formation 

After  discovery  of  a  test  and  evaluation  requirement,  the  first  challenge  in  the  NATO  T&E  process  is  to 
identify  the  end-user  community  and  appropriate  T&E  participants.  A  test  team  is  established  to  facilitate 
formal  early  tester  involvement  and  carry  out  all  T&E  planning,  execution,  and  reporting  for  the  program. 

4.6.4.3  Test  Planning  Through  Results  Reporting 

One  of  the  primary  functions  of  the  test  team  is  to  integrate  test  planning  from  initial  test  design  and  concepts 
through  to  a  Test  and  Evaluation  Master  Plan  (TEMP)  approved  by  the  NATO  leadership  or  by  the  NTO  for 
execution;  then  they  must  carry  out  the  test  and  report  the  results  so  that  decisions  may  be  made  by  the  NATO 
leadership  regarding  the  object  of  the  tests. 

4.6.5  Operational  Test  Development 

Just  as  is  the  overall  integrated  testing  process,  the  mechanism  for  development  of  the  full  operational  test  as 
part  of  that  process  is  well-defined  and  well-proven  to  be  effective.  It  has  several  stages,  each  of  which 
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necessarily  leads  to  the  next,  and  finally  to  a  reportable  result.  Though  the  contents  of  each  of  these  steps  may 
be  modified  to  reflect  the  realities  of  NATO  posture  and  equipment  to  be  tested,  a  positive  “check-off’  should 
be  made  at  each  stage  of  the  planning  process  to  ensure  that  no  important  steps  are  missed. 


Initial  Test  Design  (ITD)/Integrated  Test  Concept  (ITC) 

V 

Operational  Test  -  Test  Concept  (OT  TC) 

V 

Test  &Evaluation  Master  Plan  (TEMP) 

V 

Operational  Test  Readiness  Review  (OTRR) 


Figure  4-2:  Basic  Outline  of  the  Test  Development  Process. 


4.6.5.1  Initial  Test  Design  (ITD)  /  Integrated  Test  Concept  (ITC) 

The  ITD  starts  the  iterative  process  of  test  concept  and  test  plan  development  that  culminates  in  executable 
test  plans.  The  ITD  process  culminates  in  an  Integrated  Test  Concept  (ITC)  that  includes  an  initial  description 
of  test  scenarios,  test  locations,  exercises,  T&E  methodologies,  operational  impact  assessments  and  issues, 
and  projections  for  future  increments.  The  test  plan  fleshes  out  and  documents  the  details  that  are  known  at  the 
time  of  pre-test  planning  in  order  to  build  a  solid  basis  for  a  test  approach  and  to  communicate  that  approach 
to  others.  This  is  accomplished  by  identifying  the  battle  space  conditions  and  testing  constraints,  thereby 
leading  to  a  set  of  test  events.  Further  discussion  leads  to  a  basis  of  estimate  and  the  identification  of  resources 
(test  articles,  personnel,  etc.),  determination  of  execution  methodologies  (field  test,  modelling/simulation, 
etc.),  identification  of  test  capability  requirements  and  shortfalls,  and  refinement  of  the  operational  test 
activities  and  schedule. 

4.6.5.2  Operational  Test  -  Test  Concept  (OT  TC) 

The  OT  TC  is  a  detailed,  fleshed-out  update  of  the  approved  initial  test  design,  of  which  an  outline  is  found  at 
Annex  D.  It  includes: 

•  Review  of  requirement  documents  to  ensure  that  operational  requirements  will  be  met  (A  proposed 
method  for  reviewing  Capabilities-based  requirements  documents  is  found  at  Annex  C); 

•  Refinement  of  test  methodologies; 

•  Identification  of  evaluation  criteria; 

•  Determination  of  the  rating  methodology  for  operational  effectiveness  and  suitability; 

•  Refinement  of  the  sample  size; 
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•  Determination  of  realistic  test  scenarios; 

•  Development  of  a  plan  for  the  use  of  exercises  and  test  capabilities  needed  to  support  operational  test 
execution;  and 

•  Identification  of  testing  events  that  are  required  to  support  operational  test  conclusions. 

4.6.5.3  Test  and  Evaluation  Master  Plan  (TEMP) 

The  TEMP  will  identify  data  and  resource  requirements  to  support  the  assessment/evaluation.  It  will  list 
Measures  of  Performance  (MOP)  which  will  be  analyzed,  as  well  as  describe  test  limitations,  safety  and 
security  issues,  specific  test  events,  scenarios,  schedule,  measures,  data  collection  (who,  what,  when,  where, 
why  and  how),  reduction,  and  analysis.  It  will  show  linkages  between  data  to  be  collected,  information  to  be 
obtained,  and  conclusions  needed.  The  NTO  should  review  and  approve  the  TEMP  prior  to  the  start  of  any 
actual  testing. 

4.6.5.4  Operational  Test  Readiness  Review  (OTRR) 

The  OTRR  is  a  means  for  the  NTO  operational  test  leadership  to  review  readiness  for  dedicated  operational 
testing.  NATO  may  utilize  member  nation  test  experts  to  provide  this  service.  The  OTRR  will  address 
operational  test  posture  and  review  actions  of  contributing  agencies  in  preparation  for  test  start.  The  review 
includes  the  status  of  operational  test  readiness  certification  (see  below),  test  plan  changes,  test  team  status 
(staffing  and  training),  test  capability  status,  and  general  confidence  that  the  system  is  ready  to  begin 
operational  testing.  The  status  of  time,  cost,  resources,  and  scope  baselines  should  be  discussed.  Any  OTRR 
should  satisfy  senior  decision  makers  that: 

•  The  system  is  ready  for  the  test; 

•  Pre-test  analysis  includes  the  assurance  that  test  outcomes  will  address  objectives; 

•  Test  resources  and  personnel  are  completely  prepared  for  the  test; 

•  Known  anomalies  (e.g.,  fewer  than  the  standard  number  of  end-user  participants)  have  not  increased  the 
risks  associated  with  executing  the  test; 

•  All  reasonable  efforts  have  been  made  to  minimize  risk;  and  that 

•  Test  results  will  provide  the  appropriate  inputs  for  decision-making  regarding  the  tested  system. 

4.6.6  Test  Team  Concepts,  Tools  and  Techniques 

Special  Note:  The  functions  implied  in  this  section  do  not  necessarily  require  separate  teams,  groups, 
or  committees  but  are  discussed  separately  to  identify  discrete  functions  which  must  be  accomplished. 
The  functions  can  be  incorporated  into  presently  defined  NATO  components  (e.g.,  ACT,  NAMSA,  NTO, 
or  Expert  Groups). 

4.6.6. 1  Data  Management 

All  testers  will  establish  rigorous  data  control  and  accountability  procedures.  All  T&E  team  members  share  in 
the  responsibility  for  acquiring,  monitoring,  controlling,  and  assuring  the  quality  and  security  of  T&E  data 
supporting  the  test. 
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4.6.6.2  Timely  Data  Analysis 

The  quality  of  the  test  results  is  largely  determined  by  the  quality  of  the  test  data.  First  hand  observation  of 
test  conditions  and  execution  is  the  best  means  of  ensuring  data  are  objective  and  ultimately  useful.  Test  teams 
must  constantly  assess  if  sufficiently  accurate  data  is  being  collected  and  if  the  appropriate  analytical 
techniques  are  supporting  the  test  objectives  as  the  test  progresses.  Data  analysis  should  be  done  concurrently 
with  test  execution  if  possible. 

4.6.6.3  Timely  Release  of  Data 

All  testers  will  release  validated  test  data  and  factual  information  as  early  as  practical  to  the  NTO  or  to 
contractors  who  directly  support  NATO.  This  data  may  be  preliminary  and  if  so  should  be  clearly  identified  as 
such.  Some  types  of  data  and  test  information  that  are  evaluative  in  nature,  such  as  scoring  of  failures  or 
opinion  surveys,  can  be  withheld  until  the  evaluation  and  final  report  are  complete. 

4.6.6.4  Recording  Data  Origins 

The  origins  of  all  T&E  data  must  be  accurately  identified,  validated,  and  recorded.  Testers  may  use  data  from  a 
wide  variety  of  sources,  such  as  system  or  support  contractors,  other  member  nation  and  NATO  agencies, 
exercises  and  experiments,  and  any  prior  operational  testing.  The  goal  is  to  obtain  sufficient  amounts  of  useable 
and  credible  data  to  enhance  confidence  in  the  results  and  thus  enable  better  support  program  decisions. 

4.6.6.5  Certification  Readiness  for  Operational  Testing 

The  NTO  should  formally  certify  that  systems  are  ready  to  enter  the  dedicated  phase  of  operational  testing 
prior  to  the  actual  start  of  a  test  program.  Coordination  between  NATO  testers  and  the  NTO  is  necessary  to 
identify  and  resolve  any  concerns  about  system  maturity,  verification  of  test  assets,  resources,  facilities  and 
equipment  required  for  the  test. 

4.6.7  Contractor  Involvement  in  T&E 

Many  professional  test  organizations  place  strict  limits  on  contractor  involvement  in  T&E.  Contractors  may  be 
involved  in  the  test  process,  but  their  responsibilities  must  be  clearly  delineated  in  the  test  plan.  Operational 
testers  must  carefully  distinguish  between  two  fundamentally  different  types  of  contractors:  prime  or  system 
contractors  who  build  the  system,  and  support  contractors  who  support  NATO  in  carrying  out  its  functions. 
The  first  group  may  have  inherent  biases  regarding  the  outcome  of  the  testing,  and  may  have  vested  interests 
in  the  outcome  -  their  companies  may  stand  to  profit  if  the  tested  equipment  is  procured  for  NATO  use. 
The  second  group  may  be  considered  essentially  as  NATO  employees  for  the  purposes  of  this  testing, 
and  should  be  presumed  to  be  neutral  as  to  the  outcome  of  the  T&E  process. 

4.6.7.1  System  Contractors 

Operational  testers  must  strictly  avoid  situations  in  which  system  contractors  could  influence  or  reduce  the 
credibility  of  T&E  results,  or  could  compromise  the  realistic  and  unbiased  accomplishment  of  T&E  scenarios. 
Operational  testers  must  ensure  the  quality  and  integrity  of  any  system  contractor  data  used  for  the  integrated 
portions  of  dedicated  T&E.  The  data  must  be  accurate,  objective,  reliable,  and  available  for  independent 
evaluation. 
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4.6.7.2  Support  Contractors 

The  limitations  on  support  contractor  involvement  in  OT&E  are  distinctly  different  than  those  for  system 
contractors.  Generally  speaking,  system  contractors  will  not  be  involved  in  the  establishment  of  criteria  for 
data  collection,  performance  assessment,  or  evaluation  activities  for  the  T&E  process.  This  limitation  does  not 
apply  to  a  support  contractor  which  has  participated  in  such  development,  production,  or  testing  solely  in 
testing  or  test  support  on  behalf  of  NATO.  For  example,  a  support  contractor  working  for  NATO  on  the  same 
program  may  also  be  involved  in  the  T&E.  Support  contractors  may  be  used  to  collect  and  manage  data  in 
support  of  T&E  evaluations. 

4.6.7.3  Areas  in  Which  System  Contractors  May  Support  T&E 

System  contractor  support  in  the  T&E  process  may  be  beneficial  in  providing  logistical  support,  test  failure 
analyses,  and  software  and  instrumentation  support  which  would  increase  the  value  of  T&E  data  generated, 
while  maintaining  the  integrity  of  test  results: 

•  They  may  be  involved  in  conducting  and  reporting  analyses  of  test  failures  to  assist  in  isolating  causes 
of  failure,  but  excluding  participation  in  data  scoring  and  assessment  conferences. 

•  Their  activities  may  legitimately  include  providing  and  operating  system-unique  test  equipment, 
test  beds,  and  test  facilities  which  may  include  software,  software  support  packages,  instrumentation  and 
instrumentation  support. 

•  They  may  assist  by  providing  logistical  support  and  training  as  required  in  the  event  that  such  services 
have  not  yet  been  developed  and  are  not  available  from  NATO  or  any  national  sources. 

•  Prime  Contractors  may  be  involved  in  the  T&E  process  by  providing  data  generated  prior  to  the  conduct 
of  the  T&E,  if  deemed  appropriate  and  validated  by  the  NTO  in  order  to  ensure  that  critical  issues  are 
sufficiently  and  adequately  addressed. 

4.6.8  NATO  Member  Nation-Specific  Operational  Testing 

When  NATO  member  nation-specific  test  organizations  can  uniquely  meet  NATO’s  testing  requirements, 
the  NTO  can  request  that  that  test  organization  provide  the  operational  testing.  If  individual  nations  perform 
the  testing  on  behalf  of  NATO,  they  must  abide  by  the  same  requirements  as  would  NATO  itself  in  carrying 
out  the  testing,  as  expressed  in  this  document. 

4.6.9  NATO  Test  Organization  Support 

Per  the  NATO  Experimentation  Programme  of  Work  (EPOW),  member  nation  test  responsibilities  include 
support  of  NATO-conducted  operational  tests.  When  the  NTO  requires  support  from  a  member  nation  test 
organization,  the  level  of  support,  specific  dates  for  support,  and  a  list  of  needed  resources  should  accompany 
the  support  request  and  will  be  agreed  upon,  in  writing,  between  the  NTO  and  the  member  nations’  test 
organizations.  If  member  nations  are  providing  testing  services  for  NATO,  NATO  needs  to  approve  the  test 
plan.  National  processes  must  essentially  cover  the  NATO  requirements,  even  though  the  established  national 
processes  may  have  many  more  required  steps  and  details.  The  elements  of  the  NATO  T&E  strategy  will  be 
clearly  delineated  and  will  be  met.  Test  reports  will  include  at  a  minimum  that  data  required  of  a  NATO- 
performed  test. 
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4.7  TEST  AND  EVALUATION  RESULT  REPORTING 

A  major  function  of  testing  is  the  early  identification,  documentation,  reporting,  and  tracking  of  system 
deficiencies  and  enhancements.  NTO  has  overall  responsibility  for  establishing  and  administering  a  deficiency 
reporting  system  for  the  program,  which  will  report  in  detail  all  goals  or  activities  in  which  the  equipment 
failed  to  meet  the  test  requirements.  NTO  will  establish  procedures  for  submitting,  screening,  prioritizing, 
and  tracking  Deficiency  Reports  (DR)  from  all  sources.  An  example  of  an  Operational  Test  Final  Report 
outline  is  found  at  Annex  E. 

4.7.1  Results  from  Operational  Testing 

Specific  procedures  outlined  in  the  TEMP  will  be  used  to  create  the  DR,  and  DRs  will  be  submitted  promptly 
after  conclusion  of  the  test.  There  are  two  categories  of  DRs  which  are  described  in  the  Reference  section  of 
this  report.  NTO  will  develop  a  set  of  reports  and  official  recommendations  as  related  to  test  outcomes. 
All  Recommendations  arising  from  the  test  will  be  reported  to  ACO,  the  Committee  of  the  Chiefs  of  the 
Military  Services  in  NATO  (COMEDS)  and  the  NATO  Joint  Analysis  and  Lessons  Learned  Center  (JALLC). 
Testers  will  supply  all  available  data  to  assist  the  NTO  in  analyzing  and  fixing  problems  identified  and 
reported  in  the  DR. 
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5.1  INTRODUCTION 

One  mechanism  for  carrying  out  a  proposed  T&E  procedure  for  advanced  medical  technology  in  an  effort  to 
determine  how  this  technology  will  fit  into  (or  affect)  NATO  medical  doctrine  is  the  Medical  Technology 
Demonstration  (MTD).  MTDs  demonstrate  the  potential  exploitation  of  mature  advanced  medical  technologies 
to  solve  important  military  problems.  The  global  proliferation  of  advanced  medical  technologies  and  the 
existence  of  potential  adversaries  with  relatively  easy  access  to  these  technologies  have  increased  the  need  to 
rapidly  transition  technology  from  the  developer  to  the  user.  MTDs  are  structured  to  address  the  needs  of  the 
warfighter;  to  provide  needed  medical  capabilities,  address  identified  deficiencies,  and  reduce  costs  or 
manpower  requirements  of  medical  support  options.  Each  MTD  is  aimed  at  one  or  more  warfighting  objectives 
and  should  be  reviewed  by  the  COMEDS  and  by  AGO  prior  to  being  carried  out.  An  MTD  can  be  carried  out  as 
part  of  a  larger  formal  T&E  process,  or  can  be  done  in  a  focussed  and  more  limited  manner  as  a  stand-alone 
effort. 


5.2  OBJECTIVES  OF  MTDs 

The  objectives  of  an  MTD  are: 

•  To  conduct  meaningful  demonstrations  of  the  medical  capability; 

•  To  develop  and  test  concepts  of  medical  operations  to  optimize  military  effectiveness;  and 

•  To  prepare  to  transition  the  medical  capability  into  acquisition  without  loss  of  momentum,  if  warranted. 

5.2.1  Conduct  Meaningful  Demonstrations  of  the  Medical  Capability 

The  demonstrations  are  sized  and  structured  to  provide  a  clear  evaluation  of  the  effects  of  the  equipment  being 
tested  on  military  medical  capability.  The  user  defines  the  measures  of  effectiveness  and  measures  of 
performance  that  allow  effectiveness  and  suitability  to  be  characterized.  Data  collection  is  tailored  accordingly. 
The  quantity  of  systems  in  the  MTD  must  be  sufficient  to  provide  a  valid  assessment  of  the  medical  capability, 
or  simulations  are  used  to  expand  the  battlespace  and  forces  involved  in  the  exercise.  The  user  provides,  or  at 
least  approves,  the  planned  operational  exercises  which  typically  include  red  as  well  as  blue  forces  (Alliance  and 
Opposing  Forces).  As  a  means  of  demonstrating  the  use  of  mature  medical  technology  to  address  military 
requirements,  each  MTD  must: 

•  Evaluate  the  military  utility  and  the  ability  of  proposed  solutions  to  meet  critical  military  needs; 

•  Develop  a  robust  concept  of  operations;  and 

•  Refine  requirements. 

5.2.2  Develop  and  Test  Concepts  of  Operations 

MTDs  are  frequently  based  on  advanced  medical  technologies  which  may  permit  -  or  even  demand  -  new 
concepts  of  operation,  tactics,  and  doctrine  in  order  to  realize  their  maximum  potential.  The  MTD  provides  a 
means  to  develop,  refine,  and  optimize  these  warfighting  concepts  to  achieve  maximum  utility  and  effectiveness. 


RTO-TR-HFM-1 30 


5-1 


MEDICAL  TECHNOLOGY  DEMONSTRATIONS  (MTDs) 


ORGANIZATION 


5.2.3  Prepare  to  Transition  into  Acquisition 

A  key  goal  of  MTDs  is  to  move  into  the  appropriate  phase  of  formal  acquisition  without  loss  of  momentum, 
assuming  the  user  provides  a  positive  evaluation  of  the  medical  capability.  Each  MTD  should  have  a  clear 
acquisition  goal  covering  both  the  MTD  and  post  MTD  phases.  They  also  must  have  a  supportability  concept 
to  include  the  identification  of  areas  of  contractor  support  and  organic  support  with  the  appropriate  provisions 
for  each.  In  addition,  there  must  be:  associated  plans  for  the  development  of  formal  operational  requirements; 
documents  addressing  interoperability,  life  cycle  cost,  manning,  and  training;  and  logistics  supportability. 


5.3  FOCUS  OF  MTDS 

There  are  several  key  criteria  against  which  MTD  candidate  technologies  are  evaluated:  response  to  user 
needs,  maturity  of  these  technologies,  and  potential  effectiveness. 

5.3.1  User  Needs 

MTDs  focus  on  addressing  critical  military  medical  needs.  To  evaluate  proposed  solutions  to  meet  these  needs, 
intense  user  involvement  is  required.  MTDs  place  mature  medical  technologies  in  the  hands  of  the  user  and  then 
conduct  realistic  and  extensive  military  exercises  to  provide  the  user  an  opportunity  to  evaluate  utility  and  gain 
experience  with  the  medical  capability.  The  process  provides  the  users  a  basis  for  evaluating  and  refining  their 
operational  requirements,  for  developing  a  corresponding  concept  of  operations,  and  ultimately  for  developing  a 
sound  understanding  of  the  military  medical  utility  of  the  proposed  solution  before  a  decision  is  made  to  enter 
into  the  formal  acquisition  process.  Furthermore,  a  key  objective  of  MTDs  is  to  provide  a  residual  operational 
medical  capability  for  the  warfighter  as  an  interim  solution  prior  to  procurement. 

5.3.2  Exploit  Mature  Technologies 

MTDs  are  based  on  mature  or  nearly  mature  medical  technologies  (generally  at  TRL  Level  8  or  9,  though  lower 
TRL  levels  may  be  incorporated  in  exceptional  circumstances).  By  limiting  consideration  to  mature  medical 
technology,  the  MTD  avoids  the  time  and  risks  associated  with  technology  development,  concentrating  instead 
on  the  integration,  and  demonstration  activities.  This  approach  permits  an  early  user  demonstration  on  a  greatly 
reduced  schedule  at  low  cost. 

5.3.3  Potential  Effectiveness 

The  potential  or  projected  effectiveness  must  be  sufficient  to  warrant  consideration  as  an  MTD  or  the  medical 
capability  must  address  a  need  for  which  there  is  no  suitable  alternative  solution. 

5.3.4  Residual  Capacity 

An  additional  goal  of  MTDs  is  to  provide  a  residual  medical  capability  to  further  refine  CONOPS  and  to 
permit  continued  use  prior  to  formal  acquisition,  as  well  as  to  provide  the  ability  to  proceed  into  formal 
acquisition  for  additional  medical  capability,  if  required.  In  other  words,  medical  equipment  or  systems 
evaluated  in  an  MTD  should  be  made  available  following  the  demonstration  for  continued  operational  use  or 
further  testing  as  appropriate. 
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5.4  MTD  APPROVAL  PROCESS 

Any  NATO  command,  sub-command,  body,  or  nation  may  request  the  NTO  to  consider  carrying  out  an  MTD  in 
support  of  ACO  or  COMEDS  requirements.  Each  MTD  must  be  managed  by  ACT  or  whatever  other  body 
(NTO)  is  assigned  by  NATO  to  supervise  medical  T&E  programs.  The  actual  conduct  of  the  demonstration  may 
be  done  by  nations,  NATO  subordinate  commands  or  other  bodies  as  requested  by  ACT.  All  potential  user  and 
development  organizations  should  be  represented  during  the  MTD  planning  process.  The  incoiporation  of  an 
MTD  into  the  NATO  exercise  program  /  Experimentation  Program  of  Work  (EPOW)  must  be  requested  by  ACT 
and  approved  through  routine  NATO  processes,  as  depicted  in  Figure  5-1. 


Figure  5-1:  MTD  Development  Process. 
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5.5  MTD  INITIATION  AND  APPROVAL  PROCESS 

As  noted  above  in  Section  5.2,  the  primary  objective  of  the  MTD  process  is  to  accelerate  and  facilitate  the 
application  of  mature  advanced  medical  technologies  to  solve  important  military  problems,  as  identified  by 
ACO  or  COMEDS,  and  thereby  provide  new  operational  capabilities  which  will  make  a  difference  to  NATO 
medical  support  capabilities. 

Thus,  the  basic  form  of  an  MTD  is  represented  by  a  technology  program  or  collection  of  technology  programs 
which  are  combined  and  integrated  into  a  demonstration  carried  out  by  NATO  to  develop  or  enhance  a 
military  medical  capability.  Generically,  this  implies  identifying  significant  medical  operational  shortfalls  and 
matching  them  up  with  technology  programs  ready  to  focus  on  a  military  medical  application.  Each  proposed 
MTD  should  arise  in  response  to  an  identified  serious  deficiency  in  some  military  medical  capability  or  to  the 
presentation  of  new  medical  technology  which  is  seen  as  having  the  potential  to  improve  NATO  medical 
support,  as  perceived  and  articulated  by  ACO,  COMEDS,  or  other  NATO  body.  Another  major  MTD  goal  is 
to  promote  NATO  interoperability  to  reach  beyond  individual  national  interests  and  medical  capabilities, 
where  this  is  appropriate. 

5.6  GUIDELINE S  FOR  MTD  IMPLEMENTATION 

5.6.1  Initial  Demonstration  Plan  (IDP) 

The  Initial  Demonstration  Plan  should  clarify  the  roles  and  responsibilities  of  the  various  parties  executing  the 
MTD  and  to  provide  unambiguous  top-level  guidance  for  carrying  out  the  process.  This  document  is  envisioned 
as  an  agreement  which  defines  the  operational  medical  capability  to  be  demonstrated,  the  general  approach, 
the  agencies  responsible  for  planning  and  conducting  the  demonstration,  and  the  approximate  funding  and 
schedule. 

The  MTD  IDP  would  typically  include: 

•  The  overall  objective(s)  and  approach  of  the  MTD; 

•  Key  participants  and  their  primary  roles  and  responsibilities; 

•  The  operational  parameters  by  which  military  medical  effectiveness  is  to  be  evaluated; 

•  The  proposed  time  scale  for  the  MTD  and  potential  follow-on  operational  medical  capability; 

•  A  report  of  estimated  funding  required  for  the  MTD;  and 

•  The  directions  necessary  to  complete  the  MTD  expeditiously. 

5.6.2  Final  Demonstration  Plan  (FDP) 

The  principal  management  tool  for  the  MTD  is  the  Final  Demonstration  Plan,  the  guidelines  for  which  are 
given  in  this  document.  The  FDP  is  an  executive-level  document,  which  will  generally  be  drafted  by  the 
NATO  Testing  Organization.  The  FDP  is  a  plan;  it  is  not  intended  to  be  immutable,  as  modifications  may  be 
warranted  from  time  to  time.  However,  all  substantive  (i.e.,  schedule,  funding,  content,  objectives)  changes 
require  approval  and  documentation  by  the  NTO. 

The  MTD  FDP  provides  a  top-level  description  of  the  demonstration  with  sufficient  detail  that  the  vital 
objectives,  approach,  critical  events,  participants,  schedule,  and  funding  are  understood  and  agreed  upon  by  all 
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relevant  parties.  Measures  of  evaluation,  to  be  considered  in  addressing  both  effectiveness  and  suitability  of 
the  capability  being  evaluated,  should  be  defined. 

The  content  of  each  FDP  should  be  tailored  to  meet  the  diverse  needs  of  that  MTD.  Specific  items  which 
should  be  addressed  within  the  plan  include  the  following. 

5.6.2. 1  Objective 

An  MTD  must  demonstrate  a  proposed  solution  to  a  critical  military  medical  need.  This  section  should  include  a 
discussion  of  the  military  medical  need,  and  how  it  may  be  met  through  the  use  of  the  equipment  or  system  to  be 
tested.  The  objectives  of  the  MTD  should  be  spelled  out  in  terms  of  the  need,  the  new  or  improved  military 
medical  capabilities,  and  corresponding  concept(s)  of  operations  (CONOPS)  which  would  be  employed  with  the 
medical  capability  to  be  demonstrated. 

5.6.2.2  Overall  Approach 

A  summary  description  of  the  demonstration  is  to  be  provided  in  terms  of  the  broad  context,  the  operational 
mission,  the  scenario  in  which  the  demonstration  occurs,  the  hardware  and  software  elements  involved 
(including  those  already  within  the  force  structure  or  under  development,  as  well  as,  those  novel  elements  of 
sufficient  technical  maturity  to  be  deemed  low  risk),  the  operational  user(s)  and  their  CONOPS, 
the  physical/synthetic  location  of  the  demonstration,  the  number  of  actual  exercises  envisioned,  the  purpose 
and  scope  of  supporting  simulations,  and  the  roles  of  all  of  the  participants.  In  addition,  the  expected  time 
frame  and  overall  funding  required  to  complete  the  demonstration  phase  will  be  summarized. 

5.6.2.3  Concept  and  Technical  Approach 

A  detailed  concept  of  the  demonstration,  and  the  planned  technical  approach  to  successfully  completing  it, 
must  be  developed  early  in  the  planning  effort. 

5.6.2.4  Scenario(s)  and  Initial  Concept  of  Operations 

After  Risk  Assessment,  as  discussed  in  Chapter  6,  the  test  operational  scenario  will  have  been  chosen. 
The  operational  scenario,  or  set  of  scenarios,  which  provide  the  context  of  the  demonstration  will  be  described. 
Typically,  the  sponsoring  user(s)  will  be  responsible  for  defining  the  specific  mission  and  scenario(s)  to  be 
addressed.  The  initial  CONOPS  anticipated  for  using  the  medical  elements  in  a  military  operation  for  each 
specified  scenario  will  be  defined  and  described  by  the  user  organization,  with  the  assistance  of  the  appropriate 
T&E  manager(s)  when  necessary. 

5.6.2.5  Emerging  Technologies 

Emerging  technologies  are  hardware  and  software  technologies  which  are  not  currently  in  the  force  structure, 
and  thus  will  be  the  most  likely  candidates  for  an  MTD.  Emerging  technologies  must  be  relatively  mature 
(i.e.,  having  an  adequate  Technology  Readiness  Level  as  noted  in  Chapter  1)  with  a  high  probability  of 
providing  significant  improvements  in  military  medical  capability.  This  section  of  the  plan  will  identify  all  of 
the  emerging  medical  technologies  planned  to  be  included  in  the  demonstration  (or  sequence  of  exercises). 
For  each  emerging  technology  identified,  this  section  should  identify  the  party  responsible  for  providing  it  for 
the  demonstration,  required  dates,  anticipated  availability,  and  the  expected  top-level  evaluation 
characteristics.  Metrics  to  be  used  for  evaluation  of  the  capabilities  of  the  technology  being  evaluated  must  be 
identified  in  advance,  as  discussed  in  Chapter  7. 
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5.6.2.6  Evaluation  Characteristics 

There  are  two  general  aspects  of  military  medical  utility.  The  first  deals  with  the  question  of  how  much  the 
overall  medical  support  can  be  improved  by  the  use  of  the  evaluated  device  or  system.  This  question  can  only 
be  addressed  from  the  integrated  perspective  of  the  operational  user.  The  second  deals  with  the  issue  of  how 
effectively  the  medical  capability  under  evaluation  performs  and  how  suitable  is  it  for  use  in  military 
operations.  To  address  this  second  aspect,  it  is  important  to  define  those  evaluation  characteristics  which  will 
be  considered  to  determine  efficacy,  and  what  weight  will  be  given  to  them  if  there  is  more  than  one  of  them, 
as  discussed  in  more  detail  in  Chapter  7. 

5.6.2. 7  Risk  Management 

Risk  management  is  discussed  in  detail  in  Chapter  6,  and  thus  will  not  be  discussed  in  detail  here,  though  such 
discussions  and  decisions  must  be  reflected  in  the  FDP.  MTDs  are  intended  primarily  to  explore  operational 
effectiveness  issues  of  mature  medical  technologies;  therefore,  high  risk  is  normally  not  acceptable. 
A  discussion  of  the  risks  will  be  provided  to  ensure  that  all  parties  understand  and  accept  those  risks. 
Technologies  with  unacceptable  risk  should  not  be  included  within  an  MTD. 

5.6.2.8  Interoperability 

Interoperability  refers  to  the  ability  of  systems,  units,  or  forces  to  provide  services  to  and  accept  services  from 
other  systems,  units,  or  forces;  and  to  use  the  services  so  exchanged  to  enable  them  to  operate  effectively 
together.  It  addresses  the  interfaces  to  other  systems  and  equipment,  as  well  as  logistics  and  other  support. 
While  not  all  interoperability  issues  are  expected  to  be  resolved  during  the  MTD,  the  FDP  should  identify  the 
known  issues  and  specify  those  which  will  be  resolved.  An  approach  for  addressing  the  remaining  issues 
should  be  included  as  well. 

5.6.2.9  Equipment 

All  operational  medical  equipment  which  is  expected  to  be  involved  in  the  demonstration,  including  any 
programmed  systems  currently  in  acquisition,  will  be  described.  In  the  event  the  actual  hardware  or  the 
necessary  quantity  thereof  is  not  available,  some  equipment  may  participate  in  the  exercise(s)  via  simulation. 
The  intent  is  to  create  as  realistic  an  operational  environment  as  possible,  one  with  sufficient  scope  to  ensure 
that  the  lessons  learned  are  valid  and  can  lead  to  an  informed  user  evaluation  decision. 

5.6.2.10  Training 

It  is  envisioned  that  in  most  cases  the  emerging  medical  systems  or  technologies  will  actually  be  operated  by 
operational  forces,  rather  than  by  contractors,  and  certainly  not  by  system  contractors  (see  Section  4.6.7). 
This  implies  that  the  personnel  designated  to  conduct  the  operations  during  the  demonstration  must  receive 
adequate  training  prior  to  the  actual  exercises.  As  a  matter  of  principle,  ACT  encourages  the  maximum  use  of 
embedded  training  capability  in  all  equipment.  This  approach  could  simplify  the  task  of  training  the  operational 
forces  participating  in  the  MTD.  The  approach  for  training  (embedded  and  otherwise)  should  be  discussed. 

5.6.2.11  MTD  Managers 

The  MTD  managers,  the  individuals  or  organizations  tasked  with  actually  carrying  out  the  demonstration, 
will  provide  the  day-to-day  direction  of  the  overall  project.  They  will  prepare  and  deliver  periodic  reports  to 
the  NTO  and  other  reviewing  authorities.  The  MTD  Managers  will  bring  to  the  attention  of  the  NTO  any 
potential  deviations  from  the  FDP  for  discussion  and  resolution,  and  concomitant  modification  of  the  Plan. 
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5.6.2.12  Acquisition  and  Contracting  Strategy 

The  overall  acquisition  strategy  for  the  system  or  capability  being  demonstrated  should  be  described.  The  plan 
should  also  include  a  description  of  the  overall  contracting  strategy  for  the  MTD  and  any  intended  follow-on 
activity.  This  strategy  should  identify  each  major  contract  needed  to  execute  the  MTD,  including  a  description 
of  the  competition  plans,  contract  type,  contract  options,  and  schedule.  If  a  sole  source  contract  is  planned, 
the  contracting  strategy  should  provide  brief  rationale  and  identify  the  source.  Definition  of  support  for 
additional  field  testing  by  the  user(s)  should  also  be  provided.  There  is  a  strong  emphasis  on  streamlined 
procurement  approaches  and  industry  participants.  If  a  technology  is  to  be  provided  by  a  national  contribution 
rather  than  by  NATO  direct  acquisition,  this  strategy  must  necessarily  follow  the  prescribed  regulations  of  that 
nation. 

5.6.2.13  Critical  Events 

A  list  of  all  the  critical  events  necessary  for  successful  and  timely  completion  of  the  demonstration  will  be 
provided.  These  events  will  serve  as  decision  milestones,  with  continuation  of  the  MTD  contingent  upon 
successful  completion.  Such  events  would  typically  include  the  approval  of  the  detailed  implementation  plan, 
major  contract  awards,  critical  pre-demonstration  readiness  assessment  tests,  the  actual  demonstration(s), 
and  final  assessment  and  acquisition  recommendations. 

5.6.2.14  Residual  Operational  Medical  Capability 

MTDs  are  normally  expected  to  leave  behind  a  limited  operational  medical  capability.  In  other  words,  if  a 
demonstration  is  successful,  it  should  be  assumed  that  the  test  equipment  will  remain  available  to  NATO  for 
follow-on  use  and  operational  testing.  This  capability  should  provide  at  least  a  minimum  operational  medical 
utility  for  the  user  so  that  it  is  included  in  the  user’s  planning  and  training  activities.  The  scope  and  role  of  the 
residual  medical  capability  (e.g.,  availability  to  support  a  contingency  operation)  should  be  described,  together 
with  the  technical  support  to  be  provided  (if  any)  during  the  two  years  following  completion  of  the  MTD. 

5.6.2.15  Transition  Plan 

Although  a  detailed  transition  plan  is  not  required  at  the  initiation  of  an  MTD,  it  is  necessary  to  consider  the 
implications  of  a  decision  to  acquire  or  to  utilize  during  the  formulation  of  the  MTD.  In  addition  to  the 
development  of  the  acquisition  and  procurement  strategy  and  life  cycle  cost  estimate,  as  described  earlier, 
the  Management  Plan  should  also  address  the  assessment  of  military  medical  utility.  It  should  describe  the 
basis  and  methodology  (including  the  role  of  simulation)  for  assessing  the  military  medical  utility  (value  to 
the  warfighter).  As  a  minimum,  it  must  assign  responsibility  and  define  the  schedule  for  the  development  of 
this  methodology.  In  addition,  the  plan  should  describe  the  general  approach  proposed  for  achieving  adequate 
supportability  for  the  residual  elements  as  well  as  for  any  systems  that  are  proposed  to  enter  directly  into 
production  and  fielding  following  a  successful  MTD.  Assuming  that  the  goal  is  to  enter  the  formal  acquisition 
process  at  the  conclusion  of  the  MTD  (which  may  not  be  a  valid  assumption  in  the  NATO  environment), 
this  section  of  the  plan  should  also  identify  the  appropriate  acquisition  organization.  The  nature  of  the 
transition  may  vary  significantly  from  one  MTD  element  to  another. 

5.6.2.16  Safety/Hazards/Environmental  Assessment 

Since  the  MTD  exercises  will  likely  involve  operational  personnel  utilizing  mature  advanced  medical 
technology  systems,  some  of  which  may  be  left  behind  as  a  residual  capability,  it  is  necessary  to  assess  the 
potential  safety  hazards  which  might  arise  in  using  the  equipment,  along  with  likely  measures  to  minimize  or 
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mitigate  these  hazards,  including  adequate  pre-test  familiarization  and  training.  The  Management  Plan  should 
include  a  brief  paragraph  discussing  these  hazard/safety  issues  in  anticipation  of  a  more  detailed,  formal 
assessment  (which  may  be  required  on  a  case-by-case  basis  prior  to  actual  commencement  of  field  exercises). 
In  addition,  potential  environmental  issues  should  be  identified  and  addressed.  Chapter  6  on  Risk  Assessment 
may  provide  assistance  in  making  such  an  assessment. 

5.6.2.17  Schedule 

The  time  scale  of  an  MTD  is  typically  from  a  few  months  to  four  years,  not  including  the  two-year  follow-on 
period  for  extended  user  operational  evaluation  required  by  the  MTD  criteria  in  many  countries.  An  overall 
schedule  showing  all  the  major  milestones,  including  critical  events  will  be  provided. 

5.6.2.18  Funding 

Funding  for  the  complete  MTD  must  be  identified  and  committed  for  all  budget  years  included.  However, 
unlike  the  case  for  a  formal  acquisition  decision,  out-year  funding  beyond  the  MTD  and  its  two-year  follow- 
on  phase  need  not  be  committed.  The  funding  baseline  should  include  all  funds  required  for  completion  of  the 
MTD  and,  separately  identified,  the  funding  of  supporting  scientific  and  technical  efforts  which  are  essential 
to  the  MTD.  The  purpose  of  the  latter  is  to  identify  the  funding  required  to  assure  successful  completion  of  the 
demonstration.  In  addition,  all  relevant  prior  year  funding  with  relevance  to  the  equipment  which  is  the 
subject  of  the  MTD  (for  at  least  the  prior  three  years),  should  be  given  in  order  to  show  the  heritage  of  the 
work  being  demonstrated  and  to  make  clear  the  degree  to  which  the  effort  is  a  new  start  or  based  upon  prior 
work.  This  will  also  serve  to  identify  the  supporting  scientific  and  technical  elements  and  their  relevance  to 
the  MTD.  Only  those  supporting  elements  which  are  direct  contributors  are  to  be  included  in  the  funding 
break-out. 

5.6.2.19  Approvals 

Final  top-level  approval  for  each  MTD  must  be  given  by  COMEDS,  the  Commander(s)  of  the  principal 
sponsoring  user  organization(s)  or  ACO.  Such  approval  is  indicated  by  signature  of  the  Management  Plan. 

5.6.2.20  Endorsements 

In  addition  to  the  executive-level  approvals  listed  above,  representatives  of  all  other  planned  participants  in  the 
MTD,  including  co-sponsoring,  operational  service/support  components,  sponsors  responsible  for  providing 
military  medical  equipment  or  emerging  medical  technologies,  commanders  of  ranges  or  test  sites  required  for 
the  demonstrations,  etc.,  will  indicate  by  endorsement  their  commitment  to  providing  whatever  appropriate 
participation  and/or  support  is  identified  in  this  plan  as  their  responsibility. 

5.6.2.21  Modifications 

The  Management  Plan  is  meant  to  be  a  flexible  document;  it  should  reflect  significant  changes  in  the  MTD 
content  which  may  occur  over  the  course  of  several  years,  particularly  as  new  insights  are  gained.  Accordingly, 
it  is  expected  that  the  plan  will  be  updated  and  revised  periodically. 
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Chapter  6  -  RISK  MANAGEMENT  IN  THE  EVALUATION 
OF  ADVANCED  MEDICAL  TECHNOLOGY 
FOR  POTENTIAL  USE  BY  NATO 

6.1  INTRODUCTION 

Risk  management  is  the  systematic  application  of  policies,  procedures,  and  practices  to  the  analysis,  evaluation, 
and  control  of  risks.  A  structured  analysis  of  the  risks  involved  should  be  an  integral  part  of  any  test  and 
evaluation  program  (Chapter  4)  for  advanced  medical  equipment  or  of  a  MTD  (Chapter  5). 

Risk  management  involves  the  identification  and  description  of  hazards  and  how  they  could  potentially  occur, 
their  expected  consequences  if  they  occur,  and  estimations  or  assessments  of  the  relative  likelihood  of  such 
occurrence.  The  estimation  of  risk  for  a  given  hazard  is  a  function  of  the  relative  likelihood  of  its  occurrence 
and  the  severity  of  harm  resulting  from  its  consequences.  Following  the  identification  and  estimations  of  risk, 
risk  management  focuses  on  controlling  or  mitigating  those  risks. 


Figure  6-1:  Sample  Flowchart  Showing  Risk  Management  of  Identified  Hazards. 


The  management  of  a  single  risk  consists  of  four  steps,  risk  detection,  risk  assessment,  risk  minimization  and 
risk  communication.  However,  any  individual  medical  device/kit/system,  drug/pharmaceutical,  biologic, 
vaccine,  combination  product,  or  medical  food  will  have  multiple  risks  attached  to  it  (e.g.,  using  under 
battlefield  conditions  rather  than  in  a  fixed  hospital)  and  the  individual  risks  will  vary  in  terms  of  severity. 
Therefore,  the  concept  of  risk  management  must  also  consider  the  combination  of  information  on  multiple 
risks  with  the  aim  of  ensuring  that  the  benefits  exceed  the  risks  by  the  greatest  possible  margin  for  the 
individual  patient  and  user. 

This  chapter  describes  the  methods  and  models  related  to  risk  and  uncertainty  which  should  be  used  by  the 
participating  nations  of  NATO  and  the  PfP  in  the  evaluation  of  advanced  medical  technology  for  potential  use 
in  the  multi-national  NATO  environment. 
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6.2  SCOPE 

6.2.1  Description  of  a  Risk  Management  System 

A  risk  management  system  is  a  set  of  activities  and  interventions  designed  to  identify,  characterize,  prevent  or 
minimize  risks  relating  to  the  use  of  a  medical  device/kit/system,  drug/pharmaceutical,  biologic,  vaccine, 
combination  product,  or  medical  food,  including  the  assessment  of  the  operational  benefits  of  their  use.  It  does 
not  usually  directly  involve  an  economic  or  cost-benefit  analysis,  which  is  more  appropriately  the  goal  of  a  full 
Health  Technology  Assessment  (This  issue  is  discussed  more  fully  in  Chapter  2). 

The  aim  of  a  risk  management  system  is  to  ensure  that  the  benefits  of  a  particular  medical  device/kit/system, 
drug/pharmaceutical,  biologic,  vaccine,  combination  product,  or  medical  food  (or  any  combination  of  them) 
exceed  the  risks  by  the  greatest  achievable  margin  for  the  individual  patient  or  for  the  target  group  as  a  whole. 
This  can  be  done  either  by  increasing  the  benefits  or  by  reducing  the  risks,  but  by  its  definition  risk  management 
focuses  upon  the  risk  reduction  approach.  Nevertheless,  whenever  possible,  increases  in  benefits  should  also  be 
considered  and  the  characteristics  of  patients  most  likely  to  benefit  from  treatment  better  defined. 

6.2.2  Risk  Communication 

Risk  communication  is  an  important  step  in  risk  management  as  well  as  a  risk  minimization  activity  (also  see 
Section  4.6.5).  Patients,  users,  planners,  logisticians  and  others  who  are  involved  in  the  use  of  a  new 
technology  need  accurate  and  well  communicated  information  about  the  risks  and  benefits  associated  with 
both  the  medical  device/kit/system,  drug/pharmaceutical,  biologic,  vaccine,  combination  product,  or  medical 
food  and  the  condition  for  which  it  is  being  used,  so  that  an  informed  choice  can  be  made  about  whether  the 
item  or  system  should  be  adopted.  Product  information  in  the  form  of  the  Summary  of  Product  Characteristics 
and  Patient  Information  Leaflets  is  an  important  means  of  informing  prescribers  and  patients  about  the  risks 
associated  with  a  particular  medicine  but  additional  materials  may  be  needed.  Risk  communication  from  an 
operational  point  of  view  must  be  provided  to  the  medical  leadership  or  medical  logisticians  to  assist  them  in 
making  determinations  as  to  whether  to  procure  and  field  a  new  system  or  product.  The  mechanism  to  ensure 
this  communication  must  be  written  into  the  test  and  evaluation  plan,  and  follow-up  efforts  must  be  made  to 
ensure  that  this  evaluation  is  made  a  part  of  the  overall  test  and  evaluation  process. 

6.2.3  Risk  Minimization  Activities 

It  is  difficult  to  provide  precise  guidance  on  which  risk  minimization  activity  should  be  used  in  a  given 
situation,  as  each  safety  concern  needs  to  be  considered  on  a  case-by-case  basis. 

Direct  measurement  of  risk  minimization  should  be  employed  whenever  feasible.  Surrogate  measures  should  be 
considered  when  this  is  not  feasible,  and  such  surrogate  measures  may  be  used  to  support  interim  assessments 
whilst  awaiting  direct  risk  minimization  measurements. 

For  example,  for  measures  based  on  the  provision  of  information  to  professionals,  descriptive  studies  or 
surveys  which  assess  whether  the  information  is  being  effectively  communicated  might  be  appropriate. 
The  use  of  medical  databases  might  also  allow  direct  measures  of  how  uniformly  such  advice  was  being 
adhered  to  by  reviewing,  for  example,  concomitant  medication  or  the  results  of  laboratory  tests.  Since  such 
studies  are  likely  to  be  required  with  increasing  frequency,  the  availability  of  such  databases  will  be  an  ever 
more  important  factor  in  risk  management.  If  the  prescribing  databases  are  further  linked  to  patient  clinical 
outcome,  a  study  of  the  adequacy  of  the  prescribing  process  could  be  designed  to  evolve  over  time  into  a  full 
risk  reduction  study.  It  is  clear  that,  even  when  risks  are  of  a  type  which  can  be  directly  measured,  ethical  and 
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practical  considerations  may  prevent  prospective  comparison.  It  may  be  scientifically  difficult  to  make  direct 
comparison  between  a  situation  with  and  without  the  intervention  to  be  assessed  and  may  not  be  achievable  in 
timescales  which  allow  the  lessons  learned  to  be  used  to  improve  risk  management.  In  particular  this  will  occur 
when  risks  associated  with  long-term  exposure  or  very  rare  events  are  to  be  reduced. 

6.2.4  Ensuring  the  Effectiveness  of  Risk  Minimization  Activities 

The  definition  of  risk  management  requires  assessment  of  the  effectiveness  of  the  interventions  which  form  part 
of  the  process.  It  is  clearly  desirable  that  activities  which  may  involve  substantial  investment  of  effort  and 
resources  should  be  shown  to  achieve  the  desired  results.  In  addition,  as  a  medical  service  measure  it  is 
imperative  that  alternative  methods  be  adopted  should  a  particular  risk  minimization  strategy  prove  ineffective. 
Assessment  of  effectiveness  will  also  increase  understanding  as  to  which  activities  are  most  appropriate  in 
addressing  specific  types  of  safety  concerns  (this  topic  is  discussed  in  more  detail  in  Chapter  4). 

6.2.5  Definitions  Specific  to  Risk  Analysis 

The  terms  “uncertainty”,  “risk”  and  “hazard”  are  often  used  interchangeably,  but  they  are  not  the  same: 

•  Uncertainty  is  the  unpredictable  possibility  of  occurrence  of  an  event. 

•  Risk  can  be  defined  as  the  combination  of  the  probability  of  an  event  and  its  consequences1.  Risk  also 
can  be  defined  as  the  combination  of  the  probability  of  occurrence  of  harm  and  the  severity  of  that 
harm2. 

•  Hazard  is  the  potential  source  of  harm3. 

Harm  is  a  physical  injury  or  damage  to  the  health  of  people  or  damage  to  property  or  the  environment. 

Risk  Management  is  a  process  for  managing  the  incidence  of  risk,  as  defined  above.  In  the  safety  field  it  is 
generally  recognized  that  consequences  are  only  negative  and  therefore  the  management  of  safety  risk  is 
focused  on  prevention  and  mitigation  of  harm.  Because  risk  cannot  be  completely  eliminated,  the  risk  that 
remains  must  be  managed. 

6.2.6  Levels  of  Risk  Management 

In  every  medical  service,  risk  management  activities  broadly  take  place  simultaneously  at  different  hierarchy 
levels: 

•  Strategic  Level  -  This  level  encompasses  risk  management  functions  performed  by  ACT  or  the  NTO 
during  their  evaluation  of  advanced  technology  for  adoption  to  support  NATO  forces.  For  instance, 
the  definition  of  risks,  ascertaining  the  institution’s  risk  appetite,  formulating  strategy  and  policies  for 
managing  risks,  and  establishing  adequate  systems  and  controls  to  ensure  that  overall  risk  remain  at 
acceptable  level  and  the  rewards  compensate  for  the  risk  taken. 

•  Operational  Level  -  This  level  encompasses  risk  management  within  the  ACO  or  theatre  surgeon/ 
medical  advisor  area.  Generally  the  risk  management  activities  performed  by  middle  management  or 
units  devoted  to  risk  reviews  fall  into  this  category. 


1  ISO/IEC  Guide  73:2002. 

2  ISO/IEC  Guide  51:1999. 

3  ISO/IEC  Guide  51:1999. 
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*  Tactical  Level  -  This  level  encompasses  risk  management  within  the  Roles  1-3  facility  areas  where 
risks  are  actually  created.  Generally  the  risk  management  activities  performed  by  ‘On-the-line’ 
management  or  units  devoted  to  risk  reviews  will  fall  into  this  category. 


6.3  RISK  ANALYSIS 
6.3.1  General  Approach 

There  is  a  wide  variety  of  methods  and  models  available  for  risk  and  uncertainty  analysis  of  medical  device/ 
kit/systems,  drugs/pharmaceuticals,  biologies,  vaccines,  combination  products,  or  medical  foods.  These  include 
simple  techniques  and  sensitivity  analysis.  Each,  if  used  properly,  can  give  scientifically  sound  results,  but  first  a 
word  of  caution  -  based  on  the  collective  experience  in  risk  management  within  the  government  and  in  the 
private  sector,  it  is  fair  to  say  that  the  sophistication  and  underlying  theory  of  many  popular  models  often  far 
exceeds  the  quality  of  the  basic  data  inputs. 

There  is  simply  no  substitute  for  taking  the  time  and  effort  to  understand  the  technical  risks  and  challenges  in 
developing  and  producing  sophisticated  defence  systems.  Historical  analogies  must  be  obtained.  Information 
from  subject  matter  experts  must  be  elicited.  Risk  and  uncertainty  analysis  cannot  be  relegated  to  an  eleventh 
hour  exercise  based  on  flimsy  inputs. 

The  most  important  part  of  the  process  of  estimating  risk  and  uncertainty,  and  probably  the  most  difficult,  is  data 
collection  and  analysis.  All  variables  potentially  affected  by  risk  and  uncertainty  first  need  to  be  identified.  These 
variables  often  include  simple  ratios  and  factors  as  well  based  on  analysis. 

There  are  a  variety  of  analytic  approaches  that  are  used  by  systems  engineers.  Analytic  approaches  used  for 
investigations  including  function  and  task  analysis,  heuristic  analysis,  and  expert  reviews.  These  approaches  can 
be  applied  within  more  comprehensive  approaches  such  as  Operational  Analysis  (which  DOES  NOT  mean 
Operational  Research  in  the  military  used  form),  Analysis  of  Similar  Systems,  Failure  Modes  Effects  Analysis 
(FMEA),  Fault  Tree  Analysis  (FTA),  Critical  Incident  Technique,  Hazard  and  Operability  Studies  (HAZOP), 
and  others.  Detailed  discussion  of  these  techniques  is  beyond  the  scope  of  this  publication. 
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Figure  6-2:  A  Partial  Fault  Tree  Analysis  for  a  Pacemaker. 


6.3.2  Collecting  Data  for  Risk  Analysis 

Data  is  the  raw  material  of  risk  and  uncertainty  analysis.  It  is  critical  to  every  estimate.  Without  good,  solid 
data,  whether  based  on  historical  analogies  or  on  sound  medical  understanding  of  the  acquisition  at  hand, 
the  risk  and  uncertainty  estimate  will  be  viewed  as  merely  a  guess  or  an  opinion  of  the  analyst.  The  more  solid 
the  data,  the  better  will  be  the  estimate.  This  issue  is  further  discussed  in  Chapter  7. 

6.3.3  Modeling  and  Simulation 

Modeling  and  Simulation  (M&S)  can  be  used  as  a  tool  to  support  the  development  and  operational  use  of  new 
concepts  and  systems  for  the  future.  M&S  also  help  to  better  train  and  use  existing  forces  and  equipment  and  to 
improve  operations  in  the  new  environment.  The  simulation  of  the  Military  Decision  Process  is  a  critical  part  in 
the  use  of  M&S  for  this  purpose.  Modeling  and  Simulation  will  be  not  discussed  in  this  document4. 

6.3.4  Ergonomic  (Human  Factors)  Testing 

A  device  that  is  easy  for  one  person  to  use  safely  and  effectively  might  present  problems  for  another  person. 
Similarly,  a  device  that  is  easy  for  a  certain  group  of  users  to  use  safely  and  effectively  could  be  difficult  for 
another  group.  Users  need  devices  that  they  can  use  safely  and  effectively,  and  an  evaluation  of  this  generic 
“usability”  must  be  made  an  integral  part  of  the  T&E  process. 

A  well-designed  user  interface  will  facilitate  correct  actions  and  will  prevent  or  discourage  actions  which 
could  result  in  hazards.  The  user  interface  includes  all  components  of  a  device  with  which  users  interact  while 
using  it,  preparing  it  for  use  (e.g.,  calibration,  set-up,  unpacking),  or  performing  maintenance  (e.g.,  repairing, 
cleaning).  It  includes  hardware  features  that  control  device  operation  such  as  switches,  buttons,  and  knobs  and 
device  features  that  provide  information  to  the  user  such  as  indicator  lights,  displays,  auditory,  and  visual 


4  For  more  information  on  the  subject  of  simulation,  see  RTO-EN-017  AC/323(SAS-032)TP/26  “Simulation  of  and  for  Military 
Decision  Making”  and  the  RTO  “Simulation  Resource  Library  (SRL)”. 
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alarms.  The  user  interface  also  includes  the  logic  that  directs  how  the  system  responds  to  user  actions 
including  how,  when,  and  in  what  form  information  (feedback)  is  provided  to  the  user.  An  important  aspect  of 
the  user  interface  is  the  extent  to  which  the  logic  of  information  display  and  control  actions  is  consistent  with 
users’  abilities,  expectations,  and  likely  behaviors. 

Increasingly,  user  interfaces  for  new  medical  devices  are  computer-based.  In  these  cases,  interface 
characteristics  include:  the  manner  in  which  data  is  organized  and  presented,  control  and  monitoring  screens, 
screen  components,  prompts,  navigation  logic,  alerting  mechanisms,  data  entry  requirements,  help  functions, 
keyboards,  mice,  and  pointers.  The  size  and  configuration  of  the  device  are  important  parts  of  the  user 
interface,  particularly  for  hand-held  devices.  Device  labeling,  packaging,  training  materials,  operating 
instructions,  and  other  reference  materials  are  also  considered  part  of  the  user  interface. 

An  important  concept  pertaining  to  user  interface  use-safety  is  error  tolerance.  Error  tolerance  is  the  quality  of 
a  user  interface  which  prevents  or  mitigates  dangerous  or  disastrous  consequences  when  an  error  occurs. 
Humans  make  errors.  Some  kinds  of  error  can  be  anticipated  and  are  essentially  unavoidable  -  such  as 
inadvertently  pressing  an  adjacent  key  on  a  keypad,  or  bumping  the  keypad  inadvertently  while  doing  other 
tasks.  A  good  device  design  will  increase  the  likelihood  that  the  design  is  tolerant  of  errors  that  are  likely  to  be 
made  by  users.  There  are  many  ways  to  do  this;  one  example  is  the  placement  of  a  shield  over  the  button  that 
initiates  a  beam  of  radiation  to  prevent  inadvertent  activation.  The  logic  of  device  operation  can  also 
determine  its  degree  of  error  tolerance.  For  example,  some  devices  include  “interlocks”  or  mechanisms  that 
prevent  a  critical  process  from  being  initiated  without  users  verifying  their  intent  to  initiate  it  or  necessitating 
extra  control  steps  to  be  performed  before  proceeding.  In  other  cases,  devices  can  be  designed  to  do  tasks  that 
users  do  not  do  well,  such  as  timing  certain  steps  in  operational-testing  procedures,  remembering  set-up 
parameters,  or  test  dates,  or  performing  calculations.  For  complex  procedures,  devices  can  prompt  users  to 
perform  the  appropriate  action  at  critical  points  in  the  procedure. 

6.3.5  Operational  Testing 

Operational  testing  is  a  powerful  technique  used  to  assess  the  user’s  interaction  with  a  product.  This  technique 
can  also  be  used  to  identify  and  understand  previously  unanticipated  or  poorly  understood  use  scenarios 
resulting  in  hazards  if  care  is  taken  to  focus  on  the  safety  and  effectiveness  perspectives.  The  central 
advantage  of  operational  testing  is  that  device  use  is  realistic  and  the  results  of  the  process  are  more 
representative  of  actual  use  than  results  obtained  through  analytic  approaches. 

Operational  testing  involves  the  systematic  collection  of  data  from  users  (participants)  using  a  device 
(or  device  component)  in  realistic  situations.  Data  is  obtained  in  a  variety  of  ways,  including  user  feedback, 
manual  and  automated  measures  of  user  performance,  and  observation.  Often,  the  most  convenient  data 
collection  methods  focus  on  subjective  user  feedback.  User  feedback  includes  descriptions  by  test  participants 
of  difficulties  encountered,  good  and  bad  aspects  of  the  device  user  interface  characteristics,  including  the 
logic  of  device  operation,  and  suggested  changes.  Careful  collection  of  subjective  assessment  of  device  use 
can  identify  problems  that  were  noticed  by  test  participants  (“concerns”  or  “close  calls”)  but  which  did  not 
manifest  themselves  as  errors  during  use  and  which  were  not  identified  in  objective  performance  measures. 
(Chapter  7  discusses  this  issue  in  more  detail.) 

Objective  user  performance  measures  include  the  type  and  number  of  errors,  time  required  to  do  tasks, 
requests  for  help,  accuracy,  and  the  success  or  failure  on  individual  tasks  and  overall  performance. 
The  application  of  specific,  objective  user  performance  measures  enhances  and  focuses  subjective  user 
feedback.  Performance  measures  are  particularly  useful  for  complex  devices,  where  users  might  not  be  aware 
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of  (and  therefore  unable  to  evaluate)  potentially  hazardous  use  scenarios.  These  measures  are  also  important 
for  military-use  devices  where  users  are  often  not  aware  that  they  are  inadvertently  affecting  the  performance 
or  accuracy  of  the  device  in  some  way.  Outlier  data  from  performance  measures  is  often  informative  and 
should  be  investigated  to  determine  the  nature  and  pattern  of  the  use  scenarios  associated  with  them. 

Operational  testing  can  be  done  in  a  variety  of  ways  in  various  degrees  of  complexity  and  formality,  one 
mechanism  of  which  is  the  MTD  (See  Chapter  5).  However  it  is  done,  it  should  include  the  following: 

•  An  overall  goal  of  improving  the  usability,  including  safe  and  effective  device  use; 

•  Test  participants  represent  intended  users; 

•  Test  participants  do  real  tasks,  particularly  tasks  which  will  indicate  whether  safe  and  effective  use  is 
achieved; 

•  A  focus  on  high  risk  use  scenarios; 

•  Testers  who  observe  and  record  important  aspects  of  what  test  participants  do  and  say  (participants  can 
also  respond  to  questionnaires,  or  be  interviewed  following  the  use  of  the  device);  and 

•  Data  collected  to  support  the  identification  of  potential  use-related  hazards  and  the  development  of 
specific  recommendations  to  address  them. 

The  validity  of  testing  depends  on  the  extent  to  which  realistic  or  simulated  environments  are  used  during  the 
testing.  For  example,  in  clinical  settings  users  must  perform  multiple  tasks  simultaneously.  These  tasks  involve 
individual  devices,  multiple  devices,  and  duties  unrelated  to  device  use.  Users  must  constantly  trade-off  accuracy 
for  speed.  In  battlefield  environments,  users  might  be  distracted  or  have  medical  conditions  that  affect  their 
abilities  to  interact  with  the  device.  Users  can  also  drop  devices  or  expose  them  to  various  temperatures  and 
humidity  levels,  beyond  the  designated  design  criteria.  Clinical  and  operational  users  might  try  to  cut  costs. 
There  are  many  aspects  of  the  use  environment  that  can  affect  device  use. 

The  amount  of  thinking  and  concentration  a  person  exerts  while  using  a  device  is  called  mental  workload. 
The  mental  workload  imposed  on  users  by  the  environment  in  which  they  use  devices  can  exceed  their 
abilities  to  use  devices  properly.  For  instance,  in  an  operating  room,  there  could  be  too  many  alarms  on 
different  devices  for  an  anesthetist  to  be  able  to  identify  the  source  of  any  single  alarm.  Mental  workload  is 
often  used  synonymously  with  mental  “stress”.  There  can  be  a  physical  component  to  workload  associated 
with  medical  device  use  (physical  workload)  that  also  adds  to  the  stress  experienced  by  the  user.  Under  high 
stress  levels  -especially  under  battlefield  conditions-,  the  user  is  distracted  and  will  have  less  time  to  make 
decisions,  consider  multiple  device  outputs,  follow  complex  operating  logic,  or  physically  manipulate  device 
components.  Devices  that  can  be  used  safely  under  conditions  of  low  stress  (i.e.,  low  workload)  could  be 
difficult  or  dangerous  to  use  under  conditions  of  high  stress. 

6.3.6  Cost  and  Budget  Risk 
6.3.6.1  Cost  Risk 

The  baseline  cost  estimate  contains  modeling  uncertainty  while  the  risk-adjusted  cost  estimate  contains  both 
modeling  uncertainty  and  technical  uncertainty  and  risk.  The  risk-adjusted  probability  distribution  will  therefore 
have  a  higher  mean  value  and  a  higher  variance  than  the  baseline  estimate.  The  difference  in  mean  or  expected 
values  of  the  two  distributions  is  cost  risk.  This  value  is  usually  expressed  in  monetary  rather  than  percentage 
terms.  It  accounts  for  the  cost  impact  of  unfavourable  outcomes  in  a  major  acquisition  programme.  Aggregate 
cost  risk  can  be  allocated  to  any  cost  breakdown  structure  element,  as  appropriate. 
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63.6.2  Budget  Risk 

Budget  risk  is  the  probability  that  the  actual  cost  of  a  medical  device/kit/system,  drug/pharmaceutical,  biologic, 
vaccine,  combination  product,  or  medical  food  acquisition  program  will  end  up  exceeding  a  given  budget.  In  this 
case,  the  budget  is  set  at  the  mean  value  of  the  risk  adjusted  cost  estimate.  Since  the  budget  is  finite,  there  is  a 
certain  probability  it  will  be  exceeded.  A  low  budget  implies  a  high  probability  of  an  overrun  while  a  high 
budget  implies  a  low  probability  of  an  overrun.  It  will  be  up  to  the  decision  maker  to  decide  where  he/she  wants 
to  set  the  budget,  and  how  much  risk  he/she  is  willing  to  take  in  this  regard. 

6.3.6.3  Cost  and  Budget  Risk  for  Military  Used  Systems 

Cost  risk  and  budget  risk  are  generally  a  part  of  projects  to  introduce  available  medical  devices,  kits  and 
systems.  It  is  also  a  part  of  projects  to  produce  medical  devices,  kits  and  systems.  Cost  risks  and  budget  risks 
will  not  be  further  discussed  in  this  document5. 

6.3.7  Time  Risk 

One  risk  of  potential  significance  is  that  of  time  risk.  The  demand  for  speed  in  carrying  out  evaluations,  or  for 
speed  in  fielding  items,  may  lead  to  a  demand  to  accept  higher  risks  in  other  areas,  as  described  above. 

6.4  RISK  MINIMIZATION 

6.4.1  Requirements  of  Advanced  Medical  Technology  for  Potential  Use  by  NATO 

6.4. 1.1  General  Requirements 

Medical  device/kit,  drug/pharmaceutical,  biologic,  vaccine,  combination  product,  medical  food  must  be 
designed  and  manufactured  in  such  a  way  that,  when  used  under  the  conditions  and  for  the  purposes  intended, 
they  will  enhance  and  not  compromise  the  safety  of  patients,  or  the  safety  and  health  of  users  or,  where 
applicable,  other  persons.  Medical  devices,  kits  and  systems  must  contribute  to  providing  optimum  care  for 
the  patients  under  operational  conditions  which  apply  the  principles  of  MC  326/2  and  AJP-4.10  (A). 

As  a  part  of  a  full  operational  risk  analysis,  it  may  be  that  any  risks  which  may  be  associated  with  intended  use 
of  the  medical  device/kit,  drug/pharmaceutical,  biologic,  vaccine,  combination  product,  medical  food  constitute 
acceptable  risks  when  weighed  against  the  benefits  to  the  patient  or  the  operation  and  are  compatible  with  a  high 
level  of  protection  of  health  and  safety. 

The  evaluation  of  the  risks  of  new  technology  should  include: 

•  Reducing,  as  far  as  possible,  the  risk  of  use  error  due  to  the  ergonomic  features  of  the  device  and  the 
environment  in  which  the  device  is  intended  to  be  used  (design  for  user  and  patient  safety);  and 

•  Consideration  of  the  technical  knowledge,  experience,  education  and  training  and  where  applicable  the 
medical  and  physical  conditions  of  intended  users  (design  for  lay,  professional,  or  other  users). 

The  solutions  adopted  by  the  manufacturer  for  the  design  and  construction  of  the  devices  must  conform  to  safety 
principles,  taking  account  of  the  generally  acknowledged  state  of  the  art.  In  selecting  the  most  appropriate 
solutions,  the  manufacturer  must  apply  the  following  principles  in  the  following  order: 

5  For  more  information  see  RTO-MP-096  AC/323(SAS-036)TP/27  “Cost  Structure  and  Life  Cycle  Cost  (LCC)  for  Military 
Systems”. 
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•  Eliminate  or  reduce  risks  as  far  as  possible  (inherently  safe  design  and  construction); 

•  Where  appropriate  take  adequate  protection  measures  including  alarms  if  necessary,  in  relation  to 
risks  that  cannot  be  eliminated;  and 

•  Inform  users  of  the  residual  risks  due  to  any  shortcomings  of  the  protection  measures  adopted. 

6.4. 1.2  Requirements  for  Safety  and  Performance  of  Advanced  Medical  Technology 

To  introduce  an  available  or  produce  a  new  medical  device/kit,  drug/pharmaceutical,  biologic,  vaccine, 
combination  product,  medical  food,  it  is  necessary  to  have  an  accurate  and  complete  understanding  of  how  it 
will  be  used.  Understanding  and  optimizing  how  people  use  and  interact  with  technology  is  the  subject  of 
human  factors  engineering  (HFE).  HFE  considerations  important  to  the  development  of  medical  devices 
include  device  technology,  the  users,  the  environment  in  which  the  technology  will  be  used,  how  dangerous 
device  use  is,  and  how  critical  the  device  is  for  patient  care. 

Several  general  HFE  concepts  should  be  considered  before  proceeding  with  a  discussion  of  HFE  approaches 
in  the  context  of  risk  management. 


HF  Considerations 


Outcome 


Use 

Environment 


User 


Device 


Safe  & 
effective 


Unsafe  or 
ineffective 


Figure  6-3:  HFE  Considerations  and  Outcome. 


To  avoid  harm  the  US  and  EU  have  issued  comprehensive  directives  of  design  and  manufacturing  requirements 
for  safety  and  performance,  some  of  which  are  relevant  to  each  medical  device/kit,  drug/pharmaceutical, 
biologic,  vaccine,  combination  product,  medical  food. 

These  requirements  should  concern: 

•  Chemical,  physical  and  biological  properties. 

•  Infection  and  microbial  contamination. 

•  Manufacturing  and  environmental  properties. 

•  Devices  with  a  diagnostic  or  measuring  function. 

•  Protection  against  radiation. 

•  Requirements  for  medical  devices  connected  to  or  equipped  with  an  energy  source. 

•  Protection  against  mechanical  risks. 

•  Protection  against  the  risks  posed  to  the  patient  by  supplied  energy  or  substances. 


RTO-TR-HFM-1 30 


6-9 


RISK  MANAGEMENT  IN  THE  EVALUATION  OF 

ADVANCED  MEDICAL  TECHNOLOGY  FOR  POTENTIAL  USE  BY  NATO 


ORGANIZATION 


•  Protection  against  the  risks  posed  to  the  patient  by  devices  for  self-testing  or  self-administration. 

•  Information  supplied  by  the  manufacturer. 

•  Performance  evaluation  including,  where  appropriate,  clinical  and  operational  or  field  test  evaluation. 
Annex  B  discusses  this  issue  in  more  detail. 

6.4.1.3  Operational  Requirements  for  Potential  Use  of  Advanced  Medical  Technology  by  NATO 

Every  medical  device/kit,  drug/pharmaceutical,  biologic,  vaccine,  combination  product,  medical  food  for 
potential  use  by  NATO  must  contribute  to  providing  optimum  care  for  the  patients  under  operational 
conditions  which  apply  MC  326/2  and  AJP-4.10  (A).  To  obtain  benefits  under  operational  conditions  it  might 
be  necessary  to  accept  potential  adverse  effects.  The  minimization  of  those  potential  adverse  effects  has  to 
consider  during  all  phases  of  Test  and  Evaluation  (T&E)  or  Modeling  and  Simulation  (M&S).  Chapters  4,  5 
and  7  discuss  this  issue  in  more  detail. 

6.4. 1.4  Organizational  Requirements  for  Potential  Use  of  Advanced  Medical  Technology  by  NATO 

To  introduce  an  available  medical  device/kit,  drug/pharmaceutical,  biologic,  vaccine,  combination  product, 
medical  food,  it  is  necessary  that  NATO  have  the  organizational  structures,  the  responsibilities  of  the  managerial 
staff  and  their  organizational  authority  where  the  usability,  the  quality  of  manufacture,  logistics  and  support  of 
the  products  are  provided. 

To  field  a  new  medical  device/kit,  drug/pharmaceutical,  biologic,  vaccine,  combination  product,  medical  food, 
it  is  necessary  to  have  in  particular: 

•  The  organizational  structures,  the  responsibilities  of  the  managerial  staff  and  their  organizational 
authority  where  quality  of  design  and  manufacture  of  the  products  is  concerned; 

•  The  methods  of  monitoring  the  efficient  operation  of  the  quality  system  and  in  particular  its  ability  to 
achieve  the  desired  quality  of  design  and  of  product,  including  control  of  products  which  fail  to 
conform;  and 

•  The  procedures  for  monitoring  and  verifying  the  design  of  the  products,  including  the  corresponding 
documentation  the  inspection  and  quality  assurance  techniques  at  the  manufacturing  stage  a  Concept  of 
Operations  (CONOPS)  to  set  forth  how  it  could  be  used  in  the  environment  for  which  it  designed. 

But  it  must  be  reiterated  that  it  is  neither  the  intent  nor  is  it  within  the  scope  of  this  document  to 
recommend  or  to  build  a  fixed  organizational  structure  to  carry  out  these  functions. 
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Whether  used  in  a  T&E  process  for  a  specific  item  of  equipment,  or  in  an  MTD  process  to  evaluate  the 
benefits  of  introducing  a  medical  device  or  system  into  the  NATO  Multi-National  Medical  Care  System, 
the  Evaluation  process  must  follow  essentially  the  same  pattern,  as  shown  in  Figure  7-1. 


Figure  7-1:  The  Evaluation  Process. 


7.1  THE  EVALUATION  PROCESS 

The  evaluation  process  consists  of  three  stages,  as  noted  in  the  sections  below. 

7.1.1  Quality  Requirements  Definition 

The  purpose  of  the  initial  stage  is  to  identify  required  testable  operational  characteristics  and  possible  sub¬ 
characteristics,  in  order  to  allow  the  development  of  a  testing  matrix.  The  NTO  may  consider  at  this  stage  the 
preparation  of  a  Requirements  Document  to  ensure  inclusion  of  all  testable  components  (see  Annex  C). 

7.1.2  Evaluation  Preparation 

The  purpose  of  the  second  stage  is  to  prepare  the  basis  for  evaluation.  This  stage  consists  of  three  components, 
as  noted  in  the  sub-sections  below. 

7.1.2. 1  Metrics  Selection 

The  manner  in  which  operational  characteristics  have  been  defined  does  not  allow  their  direct  measurement, 
therefore  this  can  only  be  accomplished  by  measuring  their  attributes.  The  need  exists  to  establish  metrics  that 
correlate  to  the  characteristics  of  the  device.  Every  quantifiable  feature  of  the  device  and  every  quantifiable 
interaction  of  the  device  with  its  environment  that  correlates  with  a  characteristic  can  be  established  as  a  metric. 

7.1.2.2  Rating  Levels  Definition 

Quantifiable  features  can  be  measured  quantitatively  using  metrics.  The  result,  the  measured  value,  must  be 
interpreted  as  a  rated  value. 
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7.1.2.3  Assessment  Criteria  Definition 

To  assess  the  appropriateness  of  the  device  in  its  proposed  environment,  the  results  of  the  evaluation  of  the 
different  characteristics  must  be  summarized.  The  evaluator  has  to  prepare  a  procedure  for  this,  using, 
for  instance,  decision  tables  or  weighted  averages.  The  procedure  usually  will  include  other  aspects  such  as  time 
and  cost  that  contribute  to  the  assessment  of  appropriateness  of  the  device  in  a  particular  use  environment. 

7.1.3  Evaluation  Procedure 

The  last  step  of  the  Evaluation  Process  Model  is  refined  into  three  components,  namely  measurement,  rating 
and  assessment. 


7.1.3. 1  Measurement 

For  measurement,  the  selected  metrics  are  applied  to  the  device.  The  result  is  values  on  the  scales  of  the  metrics. 

7.1.3.2  Rating 

In  the  rating  step,  the  rating  level  is  determined  for  a  measured  value. 

7.1.3.3  Assessment 

Assessment  is  the  final  step  of  the  device  evaluation  process  where  a  set  of  rated  levels  are  summarized. 
The  result  is  a  statement  of  the  appropriateness  of  the  device.  Then  the  summarized  evaluation  is  compared  with 
the  other  aspects  such  as  time  and  cost.  Finally  managerial  decision  will  be  made  based  on  the  managerial 
criteria.  The  result  is  a  managerial  decision  on  the  acceptance  or  rejection,  or  on  the  release  or  non-release  of  the 
device. 

7.1.4  Testable  Characteristics  Identification 

If  the  efficacy  of  a  device  is  to  be  evaluated,  a  structured  set  of  testable  characteristics  is  needed,  plus  a 
method  for  evaluation  and  a  quantitative  scale  of  measures. 

Several  high-level  approaches  to  measuring  the  efficacy  of  devices  exist.  These  approaches  are  based  on  the 
idea  that  there  are  a  number  of  important  high-level  factors  of  devices  that  should  be  measured.  These  factors 
are  determined  by  lower-level  criteria  which  are  supposed  to  be  much  easier  to  measure  than  factors;  for  this 
reason,  metrics  are  proposed  as  criteria. 

Especially  for  modem  Health  Care  Devices,  it  is  not  convenient  to  look  for  a  global  efficacy  profile  (a  list  of 
values  obtained  by  assigning  a  score  to  each  sub-characteristic).  A  broad  classification  of  Health  Care  Devices 
into  five  categories  (Networking,  Archiving,  Image  Processing,  Clinical,  Administration),  according  to  their 
functionality,  allows  us  to  obtain  five  separate  profiles,  which  are  easier  to  handle  than  would  be  a  global 
value  in  the  evaluation  process.  The  profiles  result  from  the  application  of  the  model  as  a  checklist  to  define 
the  relevance  of  each  sub-characteristic  to  each  class. 

The  proposed  approach,  for  the  evaluation  of  Health  Care  Devices,  is  based  on  the  ISO/IEC  9126,  which 
describes  software  quality  as  a  function  of  six  characteristics: 

*  Functionality; 
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•  Reliability; 

•  Efficiency; 

•  Usability; 

•  Portability;  and 

•  Maintainability. 

This  decomposition  reflects  the  users’  view  and  introduces  the  concept  of  quality  in  use:  users  are  mainly 
interested  in  using  the  system,  and  evaluate  it  mostly  from  the  viewpoint  of  the  performance  and  the  service  it 
provides,  rather  than  on  the  basis  of  internal  aspects  or  the  development  process.  The  proposed  approach 
suggests  a  further  decomposition  of  each  characteristic  into  a  set  of  sub-characteristics:  these  sub-characteristics 
are  a  step  closer  to  the  quantitative,  technical  aspects  of  a  device. 


Ae  the  required 
functions  available  in 
the  device  ? 


Figure  7-2:  An  Approach  to  a  Functional  Analysis  of  Health  Care  Devices1. 


7.1.5  Functionality 

Functionality  is  a  set  of  attributes  which  bear  on  the  existence  of  a  set  of  functions  and  their  specified  properties. 
The  functions  are  those  which  satisfy  stated  or  implied  needs: 

•  Suitability:  Attributes  which  bear  on  the  presence  and  appropriateness  of  a  set  of  functions  for  specified 
tasks. 


1  ISO/IEC  9126. 
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•  Accuracy:  Attributes  which  bear  on  the  provision  of  right  or  agreed  results  or  effects. 

•  Interoperability:  Attributes  which  bear  on  the  tested  item’s  ability  to  interact  with  specified  systems. 

•  Security:  Attributes  of  a  device  which  bear  on  the  tested  item’s  ability  to  prevent  unauthorized  access, 
whether  accidental  or  deliberate. 

7.1.6  Reliability 

Reliability  is  a  set  of  attributes  which  bear  on  the  capability  of  the  device  to  maintain  its  level  of  performance 
under  stated  conditions  for  a  stated  period  of  time: 

•  Maturity:  Attributes  of  the  device  which  bear  on  the  frequency  of  failure  by  faults. 

•  Recoverability:  Attributes  of  the  device  which  bear  on  the  capability  to  re-establish  the  tested  item’s 
level  of  performance  and  on  the  time  and  effort  needed  for  this  re-establishment. 

•  Fault  Tolerance:  Attributes  of  the  device  which  bear  on  the  tested  item’s  ability  to  maintain  a  specified 
level  of  performance  in  case  of  faults  or  of  infringement  of  its  specified  interface. 

7.1.7  Usability 

Usability  is  a  set  of  attributes  which  bear  on  the  effort  needed  for  use,  and  on  the  individual  assessment  of  such 
use,  by  a  stated  or  implied  set  of  users: 

•  Learnability:  Attributes  of  the  device  which  bear  on  the  users’  effort  needed  for  learning  its 
application. 

•  Understandability:  Attributes  of  the  device  which  bear  on  the  users’  effort  needed  to  recognize  the 
logical  concept  and  its  applicability. 

•  Operability:  Attributes  of  devices  which  bear  on  the  users’  required  effort  for  operation  and  operation 
control. 

7.1.8  Efficiency 

Efficiency  is  a  set  of  attributes  which  bear  on  the  relationship  between  the  level  of  performance  of  the  device 
and  the  amount  of  resources  used,  under  stated  conditions: 

•  Time  Behaviour:  Attributes  of  the  device  which  bear  on  response  and  processing  times  and  on 
throughput  rates  in  performance  of  its  function. 

7.1.9  Maintainability 

Maintainability  is  a  set  of  attributes  which  bear  on  the  effort  needed  to  make  specified  modifications, 
including  repairs: 

•  Stability:  Attributes  of  the  device  which  bear  on  the  risk  of  unexpected  effects  of  modifications. 

•  Analyzability:  Attributes  of  the  device  which  bear  on  the  effort  needed  for  the  diagnosis  of  deficiencies 
or  causes  of  failures,  or  for  identification  of  parts  requiring  to  be  modified. 

•  Changeability:  Attributes  of  the  device  which  bear  on  the  effort  needed  for  modification,  fault 
removal  or  for  environmental  changes. 

•  Testability:  Attributes  of  the  device  which  bear  on  the  effort  needed  for  validating  the  modified  part. 
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7.1.10  Portability 

Portability  is  a  set  of  attributes  which  bear  on  the  ability  of  the  device  to  be  transferred  from  one  environment 
to  another: 

•  Installability:  Attributes  of  the  device  that  bear  on  the  effort  needed  to  install  it  in  a  specified 
environment. 

•  Replaceability:  Attributes  of  the  device  that  bear  on  opportunity  and  effort  using  it  in  the  place  of 
specified  other  system  in  the  same  environment. 

•  Adaptability:  Attributes  of  the  device  that  bear  on  the  opportunity  for  its  adaptation  to  different 
specified  environments  without  applying  other  actions  or  means  than  those  provided  for  this  purpose. 

•  Conformance:  Attributes  of  the  device  that  make  the  device  adhere  to  standards  or  conventions 
relating  to  portability. 

7.2  TABLES  OF  CHARACTERISTICS  OF  ATTRIBUTES 

The  following  tables  describe  the  above  attributes  for  each  characteristic.  The  attributes  are  classified  both  for 
procurement  and  for  operational  testing  of  a  medical  device. 

These  attributes  are  indicative  and  not  exhaustive.  The  tables  have  to  be  used  as  a  means  to  comprehend  the 
characteristics  for  the  evaluation.  The  evaluator  has  to  define  the  appropriate  attributes,  during  the  quality 
requirements  definition  process,  in  order  to  conduct  a  valid  evaluation  of  the  medical  device  or  product. 


Table  7-1:  Evaluation  of  a  Health  Care  Device  or  System  for  Procurement 


EVALUATION  OF  A  HEALTH  CARE  DEVICE  OR  SYSTEM  FOR  PROCUREMENT 

Functionality 

Suitability 

The  percentage  of  desired  functionality  that  is  actually  present  in  the  device. 

Accuracy 

The  number  of  functions  implemented  compared  to  the  functions  written  in  the 
user’s  manuals. 

Security 

Estimation  of  the  probability  that  with  a  certain  amount  of  effort  (time,  money, 
equipment)  the  device  security  measurements  will  not  be  bypassed. 

Reliability 

Fault  Tolerance 

Estimation  of  the  probability  of  maintaining  a  specified  level  of  performance  in 
case  of  faults  within  a  certain  period. 

Efficiency 

Time  Behaviour 

•  The  time  that  passes  between  the  start  and  finish  of  a  process. 

•  Number  of  information  items  which  can  be  processed  sequentially. 

•  Number  of  processing  tasks  of  a  certain  type  that  the  user  can  perform  during  a 
certain  period  with  a  certain  usage  load. 

•  Maximum  time  a  certain  internal  processing  task  occupies  with  a  certain  usage 
load. 
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EVALUATION  OF  A  HEALTH  CARE  DEVICE  OR  SYSTEM  FOR  PROCUREMENT 

Efficiency 

Time  Behaviour 

•  The  speed  of  processing  by  measuring  the  elapsed  time  between  the  beginning 
of  process  requirement  and  gaining  the  result  of  the  process. 

•  The  speed  of  processing  by  measuring  the  elapsed  time  between  the  end  of 
inquiry  or  request  to  the  device  and  the  start  of  response. 

Usability 

Leamability 

•  The  proportion  of  examples,  index  entries,  illustrations  and  tables  per 
command,  and/or  the  proportion  of  references,  chapters,  sections  and  sub¬ 
headings  per  page. 

•  The  degree  of  availability  of  reference  manuals,  on-line  user’s  manuals  and 
self-tuition  documents  such  as  operation  manuals,  grammar  reference 
materials,  installation  manual,  etc. 

•  For  a  given  range  of  functions,  the  degree  of  availability  of  materials  for  pre- 
learning  (before  use  of  the  system),  such  as  self-tuition  manuals,  automated 
tutorials  or  instructions. 

•  For  a  given  range  of  functions,  the  ratio  of  the  number  of  available  learning 
functions  to  the  number  of  necessary  learning  functions. 

•  For  a  given  range  of  functions,  the  ratio  of  the  number  of  available  help 
functions  to  the  number  of  necessary  help  functions. 

Understandability 

•  The  proportion  of  functions  which  can  be  explained  by  using  clear,  familiar 
models  to  illustrate  concepts.  This  represents  the  degree  to  which  the 
functions  and  conventions  of  a  device  are  explained  through  models  using 
familiar  concepts  from  the  everyday  world. 

•  The  proportion  of  functions  presented  to  the  users  through  demonstration. 

Operability 

•  The  operability  of  the  device  is  compared  to  the  operability  of  a  pre¬ 
determined  sample  product. 

•  The  number  of  steps  for  set-up  installation  operation  which  require  human 
interaction  (for  everyday  use). 

•  The  ratio  of  number  of  set-up  performances  available  and  the  total  number 
of  performances:  availability  of  set-up  installation  restart,  availability  of 
set-up  installation  confirmation  functions. 

•  The  average  time  required  to  set  up  the  device  or  system. 

•  The  number  of  points  at  which  users  can  make  a  pause  and  from  which  the 
installation  operation  can  be  restarted. 

•  The  number  of  types  and  amount  of  resources  including  system,  hardware, 
software  and  personnel  to  be  employed  for  installation. 

•  The  number  of  steps  for  installation  confirmation  operation  which  are 
necessary  to  validate  that  the  device  or  system  has  been  successfully 
installed  and  is  ready  to  be  used  adequately. 
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EVALUATION  OF  A  HEALTH  CARE  DEVICE  OR  SYSTEM  FOR  PROCUREMENT 

Usability 

Operability 

•  The  ratio  of  operating  commands  having  default  values  to  the  total  number 
of  operating  commands. 

•  The  proportion  of  operating  commands  having  uniform  formats,  which  are 
based  on  common-sense  and  comprehensible  rules. 

•  The  proportion  of  system  message  terms  that  are  standardized. 

•  The  proportion  of  system  messages  from  device  or  system  in  which  causes 
and  corresponding  action  are  clearly  identified  by  the  user  who  received  those 
messages. 

•  The  proportion  of  functions  for  which  operating  methods  can  be  selected  to 
correspond  to  the  user’s  level  of  skill. 

•  The  proportion  of  types  of  screen-manipulating  operations  using  common 
basic  conventions  or  patterns. 

•  The  proportion  of  input/output  screen  formats  designed  with  standardized 
formats  in  which  the  position  and  form  of  input/output  fields  are  commonly 
laid  out. 

•  The  number  of  keystrokes  of  operation  required  by  the  user  to  carry  out  the 
work. 

•  The  ratio  of  strokes  required  to  repeat  an  operation  to  the  strokes  required 
for  the  first  operation  to  perform  a  specific  task. 

•  The  ratio  of  available  guide  functions  to  the  required  ones  for  a  given  set  of 
functions. 

Portability 

Installability 

•  The  installation  effort  in  man-months. 

•  Parameter  correction  ratio  which  is  to  be  changed  by  transferring  the  device. 

•  Output  list  correction  ratio  which  is  to  be  changed  by  transferring  the  device. 

Replaceability 

Function  ratio  to  be  changed  by  transferring  the  device. 

Adaptability 

•  Mean  effort  needed  to  adapt  a  unit  volume  of  the  device  to  a  different  specified 
platform. 

•  Applicable  ratio  of  operation  manual  and  procedure  without  changing  the 
device. 

Conformance 

Conformance  ratio  in  which  the  device  is  in  conformance  with  the  laws, 
provisions  and  conventions. 

Maintainability 

Changeability 

Fulfilment  degree  of  product  documents  that  are  usable  in  the  maintenance  stage. 

Testability 

•  Effort  needed  to  test  one  unit  volume  of  the  device  with  a  certain  testing 
coverage  degree. 

•  Number  of  test  cases  that  have  to  be  made  to  test  a  unit  volume  of  the  device 
with  a  certain  testing  coverage  degree. 
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Table  7-2:  Operational  Test  of  a  Medical  Device 


OPERATIONAL  TEST  OF  A  MEDICAL  DEVICE 

Suitability 

•  The  ratio  of  functions  that  has  been  changed  during  the  operational  tests 
(change  includes  addition,  modification,  and  deletion). 

•  The  number  of  improvement  requests  for  functions  from  users  during  the 
operational  tests. 

13 

Accuracy 

The  ratio  of  incorrect  processed  transactions  to  the  total  of  presented  transactions. 

o 

o 

c 
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Interoperability 

•  The  effort  needed  to  realize  interoperability  per  unity  of  size  of 
interoperability. 

•  The  ratio  of  observed  standards  to  the  standards  made  in  the  device  or 
among  the  device. 

Security 

The  ratio  of  confidential  information  that  have  access  histories  to  all  confidential 
information. 

• 

Average  time  that  passes  between  two  failures. 

• 

The  mean  time  between  one  failure  occurrence  and  succeeding  failure 
occurrence  during  a  given  period  of  time. 

• 

The  ratio  of  number  of  faults  in  a  released  product  to  the  unit  volume  of  a 
released  product. 

Maturity 

• 

The  ratio  of  faults  corrected  up  to  a  point  of  time  to  the  estimated  number 
of  faults  initially  present  in  a  product. 

• 

The  ratio  of  number  of  faults  in  a  product  to  the  unit  volume  of  a  product 
(e.g.,  design  specification,  functionality  specification). 

-4— » 

• 

The  ratio  of  test  volume  (e.g.,  number  of  test  cases)  conducted  during  the 
development  phase  to  the  unit  volume  of  a  product  tested  and  released. 

1 

• 

The  ratio  of  amount  of  tests  actually  carried  out  up  to  a  point  in  time  to  the 
total  amount  of  tests  to  be  carried  out. 

• 

Average  time  needed  to  recover  the  damaged  device. 

• 

The  ratio  of  automatically  resolved  failures  (by  the  device)  to  the  total 
number  of  failures  for  which  automatic  recovery  would  be  preferred. 

Recoverability 

• 

• 

The  mean  time  between  occurrence  of  device  breakdown  and  completion 
of  recovery  from  breakdown. 

The  mean  time  between  occurrence  of  device  breakdown  and  completion 
of  recovery  and  restart  from  breakdown. 

• 

The  mean  time  between  completing  recovery  of  device  breakdown  and 
completing  restart  after  breakdown. 

• 

The  mean  time  between  introduction  of  a  fault  and  removal  of  the  fault. 
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• 

The  number  of  times  processing  halted  due  to  incorrect  use  within  a  certain 
period. 

>> 

• 

The  extent  to  which  it  is  possible  to  halt  processing  due  to  incorrect  use. 

1 

'S 

Fault  Tolerance 

• 

The  ratio  of  number  of  observed  breakdowns  to  the  number  of  observed 
failures  through  a  given  period  of  time. 

• 

The  ratio  of  number  of  erroneous  operations  or  inputs  detected  by  a  device 
to  the  number  of  erroneous  operations  or  inputs  conducted  during  a  given 
period  of  time. 

>> 

• 

The  average  and  maximum  time  a  user  needs  for  a  certain  processing  task, 
with  a  certain  usage  load. 

o 

c 

o 

Time  Behaviour 

• 

Average  time  a  certain  internal  processing  task  occupies  with  a  certain  usage 
load. 

w 

• 

The  amount  of  jobs  processed  in  a  unit  of  time. 

• 

The  number  of  transactions  done  in  a  unit  of  time. 

• 

The  time  an  average  end-user  from  the  target  group  needs  to  learn  to  work 
with  the  device,  plus  the  amount  of  guidance  time  needed. 

Leamability 

• 

The  proportion  of  user  claims  resulting  from  incorrect  operations  caused 
by  misunderstanding  of  the  programs  or  manuals. 

• 

The  usage  time  from  the  first  usage  until  the  usage  at  which  the  operating 
time  is  shortened  to  x/y  of  the  first  use  to  perform  the  same  specific  task. 

• 

The  ratio  of  time  required  to  learn  one  operation  for  a  specific  task  and 
operation  time. 

Usability 

Understandability 

•  The  understandability  of  the  instructions,  menus,  commands,  pictograms, 
icons,  help  information,  instructions,  manuals,  etc.,  of  the  device  as  rated 
by  the  user. 

•  Rating  of  the  readability  of  the  device  (on-screen  messages,  documents, 
pictograms,  etc.). 

• 

The  extent  to  which  the  device  presents  functionality  to  the  user  without 
hindrance,  as  judged  by  a  team  of  experts  in  this  field. 

• 

The  extent  to  which  the  device  presents  functionality  to  the  user  without 
hindrance,  as  judged  by  users  after  a  period  of  use. 

Operability 

• 

The  sum  of  time  required  for  the  installation  process,  including  preparations, 
execution  of  environment  set-up  and  verification. 

• 

Average  time  interval  between  one  operation  error  (human  error  operation) 
and  the  next. 

• 

The  time  required  for  the  operation  that  is  ultimately  reduced  and  cannot  be 
reduced  anymore  by  further  improvements. 
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OPERATIONAL  TEST  OF  A  MEDICAL  DEVICE 

Usability 

Operability 

•  The  average  time  between  the  input  of  the  system  shutdown  command  at  the 
beginning  of  shutdown  operation  and  the  time  when  shutdown  is  completed. 

•  The  ratio  of  command/data  entries  that  can  be  cancelled  to  the  total 
command/data  entries. 

•  The  ratio  of  number  of  actually  implemented  means  to  required  ones  which 
are  provided  to  emphasise  expressions  to  the  user  for  a  given  set  of  functions. 

It  is  considered  that  colour,  sound,  brightness,  and  animation  are  means  to 
emphasise  expressions. 

•  The  elapsed  time  from  the  user  input  request  or  command  to  the  nearest 
response  to  that  request  by  the  device  or  system.  The  first  response  to  the 
request  may  be  a  state  or  progress  report  expressing  that  device  or  system  is 
processing  the  request.  At  least  mean,  minimum  and  maximum  time  should 
be  measured. 

•  The  elapsed  time  from  the  current  display  on  screen  after  request  to  change, 

to  the  next  complete  display  on  screen.  At  least  mean,  minimum  and  maximum 
time  should  be  measured. 

Maintainability 

Stability 

The  ratio  of  new  faults  made  at  the  revision. 

Analyzability 

•  The  ratio  of  number  of  failures  where  users  correctly  recognized  the  fault 
positions  to  the  number  of  detected  failures  caused  by  the  faults  of  the  device 
as  a  consequence  of  the  maintainer  analysing  the  failures. 

•  Mean  time  needed  to  analyse  a  failure,  and  to  discover  any  faults  arising 
from  this  failure,  and  separate  the  positions  to  be  repaired  by  the  maintainer 
who  received  the  failure  report. 

Changeability 

•  Average  amount  of  effort  needed  to  modify  the  device,  per  unit  volume  of 
the  modification. 

•  Mean  effort  needed  to  repair  a  defect  in  the  device. 

•  Mean  time  from  receiving  the  failure  report  to  sending  the  corrected  device 
in  fault  correction. 

•  Mean  time  from  the  failure  occurrence  to  the  restoration  for  end  users. 

•  Mean  work  time  for  correcting  the  discovered  device  fault  to  be  corrected 
in  fault  correction,  in  consequence  of  analysing  the  failure. 

Testability 

•  Mean  user’s  work  time  to  verify  the  fault  correction. 

•  Mean  maintainer  work  time  to  test  the  fault  correction  after  correcting  the 
fault. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

1)  Basic  Principles 

Observed  and  Reported 
in  the  Context  of  a 

Military  Capability 
Shortfall 

Definition:  Lowest  level  of 
technology  readiness.  Maintenance 
of  scientific  awareness  and 
generation  of  scientific  and 
bioengineering  knowledge  base. 
Scientific  findings  are  reviewed 
and  assessed  as  a  foundation  for 
characterizing  new  technologies. 
Scientific  research  begins  to  be 
evaluated  for  military  applications. 
Some  Scientific  Research  begins  to 
be  translated  into  technology’s 
basic  properties.  Examples  of  R&T 
outputs  might  include  paper  studies 
of  a  technology’s  basic  properties 
and  potential  for  specific  utility. 

Hardware/Software  (HW/SW) 
technology  explored.  Basic 
theories  applied  to  IT  field 
suggest  promise.  Conceptually, 
it  may  be  possible  to  use 
Informatics  Technology 
already  at  a  higher  level  of 
readiness  to  begin  to  address 
medical  needs,  thus  being 
placed  at  TRL  Level  1  for 
biomedical  purposes. 

TRL  1  Decision 

Criterion: 

Scientific  literature  reviews 
and  initial  Market  Surveys  are 
initiated  and  assessed.  Potential 
scientific  application  to  defined 
problems  is  articulated. 

TRL  1  Decision 

Criterion: 

Scientific  literature  reviews 
and  initial  Market  Surveys 
are  initiated  and  assessed. 
Potential  scientific 
application  to  defined 
problems  is  articulated. 

TRL  1  Decision 

Criterion: 

Scientific  literature  reviews 
and  initial  Market  Surveys 
are  initiated  and  assessed. 
Potential  scientific 
application  to  defined 
problems  is  articulated. 

TRL  1  Decision 

Criterion: 

Scientific  literature  reviews 
and  initial  Market  Surveys 
are  initiated  and  assessed. 
Potential  scientific 
application  to  defined 
problems  is  articulated. 

TRL  1  Decision 

Criterion: 

Identification  of  the  potential 
medical  solution  to  mission 
need  has  been  accomplished. 
Medical  Informatics  data  and 
knowledge  representation 
issues  are  defined. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

2)  Technology  Concept 
and/or  Application 
Formulated 

Definition:  Invention  begins. 

Once  basic  principles  are  observed, 
practical  applications  can  be 
postulated.  The  application  is 
speculative  and  there  is  no 
definitive  proof  or  detailed  analysis 
to  support  the  assumptions. 

Example:  R&T  outputs  are  still 
mostly  paper  studies. 

Intense  intellectual  focus  on 
the  problem  with  generation 
of  scientific  “Paper  Studies” 
that  review  and  generate 
research  ideas,  hypothesis, 
and  experimental  designs 
for  addressing  the  related 
scientific  issues. 

Intense  intellectual  focus 
on  the  problem  with 
generation  of  scientific 
“Paper  Studies”  that 
review  and  generate 
research  ideas,  hypothesis, 
and  experimental  designs 
for  addressing  the  related 
scientific  issues. 

Intense  intellectual  focus 
on  the  problem  with 
generation  of  scientific 
“Paper  Studies”  that 
review  and  generate 
research  ideas,  hypothesis, 
and  experimental  designs 
for  addressing  the  related 
scientific  issues. 

Intense  intellectual  focus 
on  the  problem  with 
generation  of  scientific 
“Paper  Studies”  that 
review  and  generate 
research  ideas,  hypothesis, 
and  experimental  designs 
for  addressing  the  related 
scientific  issues. 

Intense  intellectual  focus  on 
the  problem  with  generation 
of  scientific  “Paper  Studies” 
that  review  and  generate 
research  ideas,  hypothesis, 
and  experimental  designs 
for  addressing  the  related 
scientific  issues. 

Hardware  and  Software 
concept  of  use  begins 
development.  Overall  system 
concepts  are  documented  by 
flowcharting  or  other  system 
descriptions. 

TRL  2  Decision 

Criterion: 

Hypothesis  (es)  generated. 
Research  plans  and/or 
protocols  are  developed,  peer 
reviewed,  and  approved. 

TRL  2  Decision 

Criterion: 

Hypothesis(es)  generated. 
Research  plans  and/or 
protocols  are  developed, 
peer  reviewed,  and 
approved. 

TRL  2  Decision 

Criterion: 

Hypothesis  (es)  generated. 
Research  plans  and/or 
protocols  are  developed, 
peer  reviewed,  and 
approved. 

TRL  2  Decision 

Criterion: 

Hypothesis  (es)  generated. 
Research  plans  and/or 
protocols  are  developed, 
peer  reviewed,  and 
approved. 

TRL  2  Decision 

Criterion: 

Medical  Informatics  data  and 
knowledge  representation 
concepts  are  defined,  peer 
reviewed,  and  approved. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

3)  Analytical  and 

Experimental  Critical 
Function  and/or 
Characteristic  Proof 
of  Concept 

Definition:  Active  research  and 
development  is  initiated.  This 
includes  analytical  studies  and 
laboratory  studies  to  physically 
validate  analytical  predictions 
of  separate  elements  of  the 
technology.  Example  R&T 
outputs  include  software  or 
hardware  components  that  are  not 
yet  integrated  or  representative  of 
final  capability  or  system. 

Basic  research,  data  collection, 
and  analysis  begin  in  order  to 
test  hypotheses,  explore 
alternative  concepts,  and 
identify  and  evaluate 
technologies  supporting  drug 
development.  Initial  synthesis 
of  counter-measure 
candidate(s)  and  identification 
of  their  sites  and  mechanisms 
of  action.  Initial 
characterization  of  candidates 
in  preclinical  studies. 

Basic  research,  data 
collection,  and  analysis 
begin  in  order  to  test 
hypotheses,  explore 
alternative  concepts,  and 
identify  and  evaluate 
critical  technologies  and 
components  supporting 
candidate  biologic/ 
vaccine  constructs 
research  and  eventual 
development  of  a 
candidate  counter¬ 
measure.  Agent  challenge 
studies  are  conducted  to 
support  models  based  on 
presumed  battlefield 
conditions.  Research-scale 
process  initiation  and 
evaluation  conducted,  as 
are  studies  to  identify 
site(s)  and  mechanism(s) 
of  action,  and  potential 
correlates  of  protection  for 
vaccines,  initial  physical/ 
chemical  characterization 
of  constructs. 

Basic  research,  data 
collection,  and  analysis 
begin  in  order  to  test 
hypothesis,  explore 
alternative  concepts,  and 
identify  and  evaluate 
component  technologies. 
Initial  tests  of  design 
concept,  and  evaluation  of 
candidate(s).  Study 
endpoints  defined.  Animal 
models  (if  any)  are 
proposed.  Design 
verification,  critical 
component  specifications 
and  tests  (if  a  system 
component,  or  necessary 
for  device  Test  & 
Evaluation)  developed. 

Basic  research,  data 
collection,  and  analysis 
begin  in  order  to  identify 
and  evaluate  components 
of  the  problem,  barriers 
to  potential  solutions, 
explore  alternative 
concepts  and  test 
hypotheses.  Initial  work  to 
define  the  approach  and 
methodological  options  is 
performed  in  analytical 
and  laboratory  models. 
Examples  include  small- 
scale  field  studies  and 
laboratory 
experimentation. 

Separate  elements  of  the 
medical  information  system 
(including  Hardware  and 
Software)  components  are 
investigated  and  modeled 
but  not  yet  integrated  or 
representative  of  final 
capability  or  system. 
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ORGANIZATION 


NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

4)  Component  and/or 
Breadboard  Validation 
in  Laboratory/Field 
Environment 

Definition:  Basic  technology 
components  are  integrated  to 
establish  that  they  will  work 
together.  This  is  relatively  low 
fidelity  compared  to  the  eventual 
system.  Examples  of  R&T  results 
include  integration  and  testing  of 
ad  hoc  hardware  in  a  laboratory/ 
field  setting.  Often  the  last  stage 
for  R&T  (funded)  activity. 

Non-GLP  laboratory  research 
to  refine  hypothesis  and 
identify  relevant  parametric 
data  required  for  technological 
assessment  in  a  rigorous  (worst 
case)  experimental  design. 
Exploratory  study  of  candidate 
drugs  (e.g.,  formulation, 
route(s)  of  administration, 
method  of  synthesis,  physical/ 
chemical  properties,  metabolic 
fate  and  excretion  or 
elimination,  and  dose  ranging). 
Candidate  drugs  are  evaluated 
in  animal  model(s)  to  identify 
and  assess  potential  safety  and 
toxicity  problems,  adverse 
events,  and  side  effects.  Assays 
to  be  used  during  non-clinical 
and  clinical  studies  in 
evaluating  candidate  drugs  are 
identified. 

Laboratory  research  (non- 
GLP)  to  refine  hypothesis 
and  identify  relevant 
parametric  data  required 
for  technological 
assessment  in  a  rigorous 
(worst  case)  experimental 
design.  Exploratory  study 
of  critical  technologies  for 
effective  integration  into 
candidate  biologic/ 
vaccine  constructs,  for 
example,  environmental 
milieu  (pH,  adjuvant, 
stabilizers  and 
preservatives,  buffers, 
etc.),  route(s)/methods  of 
administration,  proposed 
production/purification 
methods,  further  physical- 
chemical  characterization, 
metabolic  fate  and 
excretion  or  elimination, 
dose  ranging,  and  agent 
challenge  studies  for 
protection.  Candidate 
biologic/vaccine 
constructs  are  evaluated  in 
animal  model(s)  to 
identify  and  assess  safety 
and  toxicity,  biological 
effects,  adverse  effects, 
and  side  effects.  Assays, 
surrogate  markers,  and 
endpoints  to  be  used 

Non-GLP  laboratory 
research  to  refine 
hypothesis  and  identify 
relevant  parametric  data- 
required  for  technological 
assessment  in  a  rigorous 
(worst  case)  experimental 
design.  Exploratory  study 
of  candidate  device(s)/ 
systems  (e.g.,  initial 
specification  of  device, 
system,  and  sub-systems). 
Candidate  devices/ 
systems  are  evaluated  in 
laboratory  and/or  animal 
models  to  identify  and 
assess  potential  safety 
problems,  adverse  events, 
and  side  effects, 

Procedures  and  methods 
to  be  used  during  non- 
clinical  and  clinical 
studies  in  evaluating 
candidate  devices/systems 
are  identified. 

Laboratory  research  to 
refine  and  test  hypotheses 
and  identify  relevant 
parametric  data  required 
for  assessment  of  candidate 
solution  sets  in  a  rigorous 
experimental  design. 
Candidate  solutions  are 
characterized  and 
evaluated,  in  laboratory 
models  to  identify  and 
assess  potential  problems 
and  verify  potential  utility. 
Procedures  and  methods  to 
be  used  in  further 
validation  of  knowledge- 
ware  candidates  are 
identified. 

Prototypes  of  components 
are  produced.  System 
components  are  integrated 
to  establish  that  the  pieces 
will  work  together.  This  is 
relatively  “low  fidelity” 
compared  to  the  eventual 
system. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

4)  Component  and/or 
Breadboard  Validation 
in  Laboratory/Field 
Environment  (cont'd) 

during  non-clinical  and 
clinical  studies  to  evaluate 
and  characterize  candidate 
biologic/vaccine  constructs 
are  identified. 

TRL  4  Decision 

Criterion: 

Proof-of-Concept 
demonstrated  for  candidate 
drug  formulations  and  animal 
models  defined. 

TRL  4  Decision 

Criterion: 

Proof-of-concept 
demonstrated  for 
candidate  biologic/ 
vaccine  constructs  and 
animal  models  defined. 

TRL  4  Decision 

Criterion: 

Proof-of-concept 
demonstrated  for  candidate 
devices/ systems  and 
laboratory/animal  models 
defined.  Initial  device 
master  record  completed. 

TRL  4  Decision 

Criterion: 

Proof-of-concept 
demonstrated  for 
knowledge-ware 
candidates,  experimental 
models,  and  methodologies 
defined. 

TRL  4  Decision 

Criterion: 

Medical  Informatics  Data 
and  knowledge  representation 
models  are  initiated  with 
representative  data  or 
knowledge  from  applicable 
domain. 
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ORGANIZATION 


NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

5)  Component  and/or 
Breadboard  Validation 
in  a  Relevant  (Operating) 
Environment 

Definition:  Fidelity  of  sub-system 
(breadboard)  representation 
increases  significantly.  The  basic 
technological  components  are 
integrated  with  reasonably 
realistic  supporting  elements  so 
that  the  technology  can  be  tested 
in  a  simulated  operational 
environment.  Examples  include 
high  fidelity  laboratory/field 
integration  of  components.  Rarely 
an  R&T  (funded)  activity  if  it  is  a 
hardware  system  or  device  of  any 
magnitude  or  system  complexity. 

Intense  period  of  non-clinical 
studies  involving  parametric 
data  collection  and  analysis  in 
well-defined  systems  and  a 
GLP/GMP  environment,  with 
pilot  lot  scale  production  and 
further  development  of 
selected  candidate(s).  Conduct 
GLP  safety  and  toxicity 
studies  in  animal  model 
systems.  Identify  endpoints  of 
clinical  efficacy  or  its 
surrogate.  Conduct  studies  to 
evaluate  the  pharmaco¬ 
kinetics  and  pharmaco¬ 
dynamics  of  candidate  drugs. 
Product  Development  Plan 
drafted. 

Intense  period  of  non- 
clinical  and  pre-clinical 
research  studies  involving 
parametric  data  collection 
and  analysis  in  well- 
defined  systems  with  pilot 
lots  of  candidate  biologies/ 
vaccines  produced  and 
further  development  of 
selected  candidates. 

Research  results  support 
proposing  a  potency  assay, 
proposing  a  manufacturing 
process  amenable  to  GMP- 
compliant  pilot  lot 
production,  identifying  and 
demonstrating  proof-of- 
concept  for  a  surrogate 
efficacy  marker  in  an 
animal  model,  applicable 
to  predicting  protective 
immunity  in  humans,  and 
demonstrating  preliminary 
safety  and  efficacy  against 
an  aerosol  challenge  in  a 
relevant  animal  model. 
Conduct  GLP  safety  and 
toxicity  studies  in  animal 
model  systems.  Identify 
endpoints  of  clinical 
efficacy  or  its  surrogate  in 
animal  models  that  may  be 
applicable  to  predicting 
protective  immunity  in 
humans.  Conduct 
studies  to  evaluate 

Further  development  of 
selected  candidates(s). 
Devices  compared  to 
existing  modalities  and 
indication  for  use  and 
equivalency  demonstrated 
in  model  systems. 

Examples  include  devices 
tested  through  simulation 
in  tissue  or  organ  models, 
or  animal  models  if 
required.  All  component 
suppliers/vendors  are 
identified  and  qualified. 
Vendors  for  critical 
components  audited  for 
GMP  compliance. 
Component  tests, 
component  drawings,  and 
device  master  record 
verified.  Product 
development  Plan  drafted. 

Candidate  knowledge- 
ware  products  (system  and 
sub-system  components) 
are  prototyped  and  tested 
in  experimental  models. 
Proto-type  components 
are  refined,  specified,  and 
tested.  Knowledge-ware 
technology  development 
plan  drafted. 

First  technical  test  of 
prototype/information  system 
components  are  integrated  and 
realistic  supporting  elements 
are  employed  so  that  the 
system  can  be  tested  in  a 
simulated  environment.  Actual 
interfaces  to  supporting 
systems  are  specified  and 
development  begins. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

5)  Component  and/or 
Breadboard  Validation 
in  a  Relevant  (Operating) 
Environment  (cont'd) 

immunogenicity,  as  well 
as  pharmaco-kinetics  and 
pharmaco-dynamics  when 
appropriate. 

TRL  5  Decision 

Criterion: 

Dependent  upon  national  legal 
requirements.  In  US,  this  is 
the  stage  at  which  an 
Investigational  New  Drug 
request  should  be  submitted 
to  regulatory  authorities  to 
determine  if  clinical  trials 
may  proceed. 

TRL  5  Decision 

Criterion: 

A  decision  is  made  at 
which  it  is  determined  that 
sufficient  data  on  the 
candidate  biologic/vaccine 
exists  to  justify 
preparation  of  a  technical 
data  package  which 
supports  continued 
development.  Dependent 
upon  national  legal 
requirements.  In  US,  this 
is  the  stage  at  which  an 
Investigational  New  Dmg 
request  should  be 
submitted  to  regulatory 
authorities  to  determine 
if  clinical  trials  may 
proceed. 

TRL  5  Decision 

Criterion: 

Regulatory  authorities  have 
reviewed  submissions  of 
required  data  to  determine 
if  clinical  trials  may 
proceed. 

TRL  5  Decision 

Criterion: 

Effectiveness  of 
knowledge-ware 
candidates  is  demonstrated 
in  a  high  fidelity 
laboratory  or  simulated 
operational  environment. 
Knowledge- ware 
technology  development 
plan  is  reviewed  and 
approved. 

TRL  5  Decision 

Criterion: 

Medical  Informatics  data  and 
knowledge  representation 
models  are  implemented  as 
data  and/or  knowledge 
management  systems  and 
tested  in  a  laboratory 
environment. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

6)  System/Sub-System 

Model  or  Prototype 
Demonstration  in  a 
Realistic  (Operating) 
Environment  or  Context 

Definition:  Representative  model 
or  prototype  system,  which  is  well 
beyond  the  representation  tested 
for  TRL  5,  is  tested  in  a  more 
realistic  laboratory  or  simulated 
operational  environment. 

Represents  a  major  step  up  in  a 
techno  logy’s  demonstrated 
readiness.  Examples  include 
testing  a  prototype  in  a  high 
fidelity  laboratory/field 
environment  or  in  simulated 
operational  environment. 

First  clinical  trials  (US  “Phase 
1”)  conducted  to  show  safety 
of  candidate  drug  in  a  small 
number  of  human  subjects 
under  carefully  controlled  and 
intensely  monitored  clinical 
conditions.  Evaluation  of 
pharmaco-kinetic  and 
pharmaco-dynamic  data  to 
support  the  design  of  well- 
controlled,  scientifically  valid 
Next-Phase  (US  “Phase  2”) 
studies.  Production 
technology  demonstrated 
through  production-scale 

GMP  plant  qualification. 

National  drug  safety 
regulators  must  evaluate 
and  approve  first  Phase 
clinical  trials  to 
demonstrate  safety  of 
candidate  medications  in  a 
small  number  of  subjects. 
Evaluation  of 
immunogenicity  and/or 
pharmaco-kinetics  and 
pharmaco-dynamics  data 
to  support  design  of 
second  Phase  clinical 
trials.  Surrogate  efficacy 
models  are  validated. 

First  phase  clinical  trials 
conducted  to  demonstrate 
the  safety  of  the  candidate 
device  in  a  small  number 
of  humans  under  carefully 
controlled  and  intensely 
monitored  clinical 
conditions.  Validation  of 
the  master  plan  for  critical 
components  and  final 
device  assembly. 

Production  technology 
demonstrated  through 
production-scale  GMP 
plant  qualifications. 

System  and  sub-system 
components  of  the 
candidate  systems  are 
configured  into  advanced 
prototypes.  Initial 
controlled  trials  with 
prototype  candidates  are 
conducted  in  validated 
model  systems. 
Observational  field  studies 
conducted  in  a  training 
environment  are  designed 
to  provide  statistical 
significance. 

Advanced  technical  testing  of 
prototype  information 
systems,  to  include  interfaces 
to  actual  supporting  systems, 
is  tested  in  a  relevant  or 
simulated  operational 
environment.  Output  is  final 
prototype. 

TRL  6  Decision 

Criterion: 

Data  from  first  phase  trials 
meet  clinical  safety 
requirements  and  support 
proceeding  to  the  next  phase 
of  clinical  studies. 

TRL  6  Decision 

Criterion: 

Data  from  first  phase 
clinical  trials  meet  clinical 
safety  requirements  and 
support  proceeding  to  the 
next  stage  of  clinical 
trials. 

TRL  6  Decision 

Criterion: 

Data  from  first  phase  trials 
meet  national  clinical 
safety  requirements  and 
support  proceeding  to  next 
phase  of  clinical  studies. 

TRL  6  Decision 

Criterion: 

Effectiveness  of  the 
product  supported  by 
controlled  studies  in 
humans  or  appropriate 
surrogate  models.  These 
studies  are  designed  to 
provide  statistical 
significance.  Validation 
protocols  designed  to  test 
prototypes  in  actual 
operational  environments 
with  humans  or  validated 
surrogate  models  are 
approved. 

TRL  6  Decision 

Criterion: 

Medical  Informatics  data  and 
knowledge  management 
systems  are  tested  with  target 
applications  in  a  relevant  or 
simulated  operational 
environment.  Configuration 
management,  administration, 
and  maintenance  issues  are 
defined. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

7)  System  Prototype 
Demonstration  in  an 
Operational  Environment 
or  Context  (e.g.,  Exercise) 

Definition:  Prototype  near  or  at 
planned  operational  system  level. 
Represents  a  major  step  up  from 
TRL  6,  requiring  the 
demonstration  of  an  actual  system 
prototype  in  an  operational 
environment,  such  as  in  a  relevant 
platform  or  in  a  system-of- 
systems.  Information  to  allow 
supportability  assessments  is 
obtained.  Examples  include 
extensive  testing  of  a  prototype 
in  a  test  bed  vehicle  or  use  in  a 
military  exercise.  Not  R&T 
funded  although  R&T  experts 
may  well  be  involved. 

Second  stage  clinical  trials  are 
conducted  to  demonstrate 
initial  efficacy  and  capture 
further  safety  and  toxicity 
data.  Product  activity 
(e.g.,  preliminary  evidence 
of  efficacy)  is  determined. 
Product  final  dose,  dose 
range,  schedule,  and  route  of 
administration  are  established 
from  clinical  pharmaco¬ 
kinetic  and  pharmaco¬ 
dynamic  data.  Approval 
sought  to  progress  to  third 
stage  of  clinical  trials. 

Second  phase  safety  and 
immunogenicity  trials  are 
conducted.  Product 
immunogenicity  and 
biological  activity 
(e.g.,  preliminary  evidence 
of  efficacy)  is  determined. 
Product  final  dose,  dose 
range,  schedule,  and  route 
of  administration  is 
established  from  vaccine 
immunogenicity  and 
biologic  activity,  and 
when  necessary  clinical 
pharmaco-kinetics  and 
pharmaco-dynamics  data. 
Second  phase  clinical 
trials  are  completed.  Data 
is  collected,  presented, 
and  discussed  with 
regulatory  agencies. 
Continue  development  is 
supported  by  regulatory 
agencies.  Clinical 
endpoints  and/or  surrogate 
efficacy  markers  and  test 
plans  are  agreed  by 
regulatory  agencies. 

Second  phase  clinical 
effectiveness  and  safety 
trials  are  conducted  with  a 
fully  integrated  device 
prototype  in  an 
operational  environment. 
Continuation  of  closely 
controlled  studies  of 
effectiveness  and 
determination  of  short¬ 
term  adverse  events  and 
risks  associated  with  the 
candidate  product. 
Functional  testing  of 
candidate  devices  is 
completed  and  confinned, 
resulting  in  final  selection 
of  prototype  device. 
Regulatory  agencies  have 
approved  continued 
development  and  testing. 

Small-scale  validation 
trials  conducted  to  test  and 
evaluate  a  fully  integrated 
prototype  in  actual 
operational  environments 
with  humans  or  validated 
surrogate  models. 

Prototype  system  is  near  or  at 
planned  operational  system. 
Actual  system  prototype  is 
demonstrated  in  an 
operational  environment 
with  end-users. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

7)  System  Prototype 
Demonstration  in  an 
Operational  Environment 
or  Context  (e.g.,  Exercise) 
(cont’d) 

TRL  7  Decision 

Criterion: 

Second  Phase  clinical  trials 
completed.  Data  collected, 
presented,  and  discussed 
with  regulatory  agencies. 
Regulatory  agencies  agree  to 
continued  development. 

Clinical  endpoints  and  or 
surrogate  efficacy  markers 
and  test  plans  are  agreed. 

Third  phase  clinical  study 
plan  is  approved. 

TRL  7  Decision 

Criterion: 

Next  phase  clinical  study 
plan  or  surrogate  test  plan 
has  been  approved. 

TRL  7  Decision 

Criterion: 

Second  Phase  clinical 
effectiveness  and  safety 
trials  are  completed. 

Final  product  design  is 
validated,  and  final 
prototypes  and/or  initial 
commercial  scale  devices 
are  produced.  Data  is 
collected,  presented,  and 
discussed  with  regulatory 
agencies.  Agencies 
support  continued 
development.  Clinical 
endpoints  and  test  plans 
agreed  to  by  regulatory 
agencies.  Next  Phase 
clinical  study  plan  is 
approved. 

TRL  7  Decision 

Criterion: 

Small-scale  validation 
trials  successfully 
completed.  Data  has  been 
collected  and  validate  the 
fully  integrated  prototypes 
and  support  proceeding  to 
full-scale  validation  trials. 

TRL  7  Decision 

Criterion: 

Medical  informatics  data  and 
knowledge  management 
systems  are  operationally 
integrated  and  tested  with 
target  applications  in  an 
operational  environment. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

8)  Actual  System  Completed 
and  Qualified  through 

Test  and  Demonstration 

Definition:  Technology  has  been 
proven  to  work  in  its  final  form 
and  under  expected  conditions.  In 
almost  all  cases,  this  TRL 
represents  the  end  of 
Demonstration.  Examples  include 
test  and  evaluation  of  the  system 
in  its  intended  environment  to 
determine  if  it  meets  design 
specifications,  including  those 
relating  to  supportability. 

Implementation  of  expanded 
controlled  and  uncontrolled 
Phase  3  clinical  trials  or 
surrogate  tests  to  gather 
information  relative  to  the 
safety  and  effectiveness  of  the 
candidate  drug.  Trials  are 
conducted  to  evaluate  the 
overall  risk-benefit  of 
administering  the  candidate 
product,  and  to  provide  an 
adequate  basis  for  drug 
labeling.  Process  validation 
completed  and  followed  by  lot 
consistency/reproducibility 
studies.  Stability  studies 
initiated.  New  Drug 

Application  submitted  to 
appropriate  national  or 
international  regulatory 
authorities. 

Implementation  of 
expanded  controlled  and 
uncontrolled  Phase  3 
clinical  trials  or  surrogate 
tests  to  gather  information 
relative  to  the  safety  and 
effectiveness  of  the 
candidate  biologic/ 
vaccine.  Trials  are 
conducted  to  evaluate  the 
overall  risk-benefit  of 
administering  the 
candidate  product,  and  to 
provide  an  adequate  basis 
for  product  labeling. 

Process  validation 
completed  and  followed 
by  lot  consistency/ 
reproducibility  studies. 
Stability  studies  initiated. 

Implementation  of 
expanded  controlled  and 
uncontrolled  Phase  3  trials 
to  gather  information 
relative  to  the  safety  and 
effectiveness  of  the 
device.  Trials  are 
conducted  to  evaluate  the 
overall  risk-benefit  of 
using  the  device,  and  to 
provide  an  adequate  basis 
for  product  labeling. 

Process  validation 
completed  and  followed 
by  lot  consistency/ 
reproducibility  studies. 

Full-scale  validation  and 
beta-testing  trials 
conducted  to  test  and 
evaluate  a  fully  integrated 
prototype  in  as  close  to  the 
intended  operational 
environment  as  possible 
(e.g.,  ATD,  ACTD,  or 
EPOW  Experiment). 
Refinement  of  product  to 
address  any  deficiencies 
revealed  in  full-scale 
validation  studies  and 
beta-test  feedback  from 
user.  Product  undergoes 
independent,  external 
review. 

Technical  testing  of  Final 
Product.  System  has  been 
proven  to  work  in  its  final 
form  and  under  expected 
conditions. 

TRL  8  Decision 

Criterion: 

Approval  of  New  Drug 
Application  by  national  or 
international  regulatory 
authorities. 

TRL  8  Decision 

Criterion: 

Approval  for  production 
given  by  national  or 
international  regulatory 
authorities. 

TRL  8  Decision 

Criterion: 

Approval  of  the  device  by 
national  or  international 
regulatory  authorities. 

TRL  8  Decision 

Criterion: 

External  review  approval 
equivalent  to  ANSI  or 
ANSI-like  standard  is 
obtained. 

TRL  8  Decision 

Criterion: 

Developmental  test  and 
evaluation  of  the  system  in 
its  intended  environment 
demonstrate  it  meets  design 
specifications.  Fully 
integrated  and  operational 
Medical  Informatics  Data  and 
knowledge  management 
systems  are  validated  in 
several  operational 
environments. 
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NATO  Technology 
Readiness  Level 
and  Definition 

Pharmaceutical 

(Drug) 

Pharmaceutical 

(Biologics/Vaccines) 

Medical  Devices 

Medical 

Knowledge- W  are 

Medical  Information 
Technology  and 
Medical  Informatics 

9)  Actual  System 

Operationally  Proven 
through  Successful 

Mission  Operations 

Definition:  Application  of  the 
technology  in  its  final  form  and 
under  mission  conditions,  such  as 
those  encountered  in  operational 
test  and  evaluation  and  reliability 
trials.  Examples  include  using  the 
final  system  under  operational 
mission  conditions. 

Post-Marketing  studies 
(clinical  or  non-clinical) 
may  be  required  by  national 
or  international  regulatory 
authorities.  Post-Marketing 
Surveillance. 

Post-Marketing  studies 
(clinical  or  non-clinical) 
may  be  required  by 
national  or  international 
regulatory  authorities. 
Post-Marketing 

Surveillance. 

Post-Marketing  studies 
(clinical  or  non-clinical) 
may  be  required  by 
national  or  international 
regulatory  authorities. 
Post-Marketing 

Surveillance. 

Final  product  and  all 
documentation  completed 
and  provided  to  end-user 
community  for 
implementation.  Examples 
include  integration  of 
infonnation  into  doctrine, 
transfer  of  information  to 
project  manager  for  use  in 
system  design. 

Operational  testing  of  the 
product.  The  system  is  in  its 
final  form  and  under  mission 
conditions,  such  as  those 
encountered  in  operational 
test  and  evaluation.  Medical 
infonnatics  knowledge 
maintenance  and  verification 
of  data  integrity  are  ongoing 
requirements  for 
transportation,  handling, 
storage,  etc. 

TRL  9  Decision 

Criterion: 

None  -  continue 
surveillance. 

TRL  9  Decision 

Criterion: 

None  -  continue 
surveillance. 

TRL  9  Decision 

Criterion: 

None  -  continue 
surveillance. 

TRL  9  Decision 

Criterion: 

Final  acceptance  of 
knowledge-ware  design 
by  end-user. 

TRL  9  Decision 

Criterion: 

Product  successfully  used 
during  military  mission  as 
part  of  operational  testing 
and  evaluation.  Logistical 
demonstration  successfully 
conducted. 

Produced  by  RTG-130.  Derived  in  large  part  from  work  done  by  NATO  Undersea  Research  Center,  United  States  National  Aeronautics  and  Space  Administration,  and 
United  States  Army  Medical  Research  and  Materiel  Command. 

Definitions  -  Only  applicable  to  Drug,  Biological,  and  Vaccine  Development: 

•  First  Phase  Studies  -  Small  dose  studies  using  10  -  50  healthy  volunteers  to  monitor:  Adverse  Events  (AE);  toxicology;  and  pharmaco-kinetics.  The  goal  is  to 
determine  safety  and  dose  ranges  for  future  trials. 

•  Second  Phase  Studies  -  Larger  studies  using  variable  dosages  on  100  -  200  target  population  patients,  to  monitor:  effect(s)  on  disease;  dosage;  safety/ AE/side 
effects;  and  pharmaco-kinetics.  The  goal  is  to  determine  appropriate  dosages  and  frequency  of  administration,  and  whether  to  proceed  with  the  development. 

•  Third  Phase  Studies  -  Still  larger  studies  at  many  sites  with  a  large  and  diverse  patient  population  to  monitor  the  effects  on  many  types  of  diseases  and  safety. 
Placebo  comparison  will  be  used,  as  will  comparison  to  other  products  on  the  market.  The  goal  is  to  determine  appropriate  clinical  usage  guidelines  and  labeling. 

NOTE:  If  a  newly  evaluated  device  or  system  seems  to  fall  into  two  or  more  of  the  above  categories  of  product,  e.g.,  a  medical  device  which  is  an  integral  part  of  a  medical 
information  system.  It  should  be  evaluated  in  accordance  with  the  rules  for  both  categories,  and  will  be  considered  to  be  at  the  lower  of  the  two  TRLs  so  determined. 
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Annex  B  -  NATO  MEDICAL  PRODUCT  REGULATORY 
CONSIDERATIONS  AND  CHECKLIST 


In  most  cases  procurement  of  medical  devices  or  products  for  NATO  use  will  be  the  responsibility,  as  it  has 
been  in  the  past,  of  a  participating  member  nation  for  its  constituent  forces.  In  these  cases,  regulatory  concerns 
are  the  responsibility  of  the  nation  rendering  medical  care  as  prescribed  by  its  countries  laws  and  regulatory 
bodies.  In  the  future,  NATO  may  seek  to  procure  medical  devices  or  products  for  use  by  multiple  member 
nations.  In  these  instances,  the  following  discussion  may  guide  a  team  planning  to  test  and/or  procure  medical 
products  to  recognize  potential  regulatory  considerations. 

B.l  MEDICAL  PRODUCT  REGULATORY  OVERSIGHT 

Procurements  covered  by  this  document  will  generally  only  include  approved  EU  (CE  mark)  or  FDA  medical 
products.  It  is  conceivable  that  approvals  by  other  regulatory  bodies  from  host  nations  could  be  considered, 
but  these  products  would  need  to  be  analyzed  in  conjunction  with  CE,  FDA,  or  alternative  consensus 
standards  to  be  developed.  Any  such  decisions  should  also  address  any  applicable  NATO  policy.  Since  forces 
from  member  nations  serve  under  NATO  command,  NATO  member  nations  should  establish  a  formal  policy 
that  medical  products  procured  and  utilized  within  the  framework  of  any  NATO  program  are  considered 
consistent  with  the  medical  product  regulatory  expectations  and  standards  of  care  of  the  member  nations. 

In  other  words,  a  policy  should  be  established  that  allow  NATO  medics  from  the  US  to  use  CE  mark  medical 
products  on  NATO  or  Partnership  for  Peace  soldiers,  sailors,  or  airman  and  all  other  categories  of  patients  if  the 
medic  is  serving  on  a  NATO  mission.  Likewise,  NATO  medics  from  other  member  or  partner  nations  could  use 
FDA  approved  medical  products  on  soldiers,  sailors,  or  airman  and  all  other  categories  of  patients  from  NATO 
or  Partnership  for  Peace  countries. 

B.2  STRATEGY  FOR  PROCUREMENT 

B.2.1  NATO  Medical  Procurement 

NATO  Medical  Procurement  will: 

•  Apply  unified  standards  of  acceptable  quality,  safety  and  efficacy. 

•  Comprehensively  evaluate  the  quality,  safety  and  efficacy  of  medical  products,  based  on  information 
submitted  by  the  manufacturers,  and  inspection  of  the  corresponding  manufacturing  sites,  along  with 
military  specific  specifications. 

•  Establish  points  of  contact  with  national  regulatory  authorities  to  assure  that  the  medical  products 
being  procured  meet  appropriate  compliance  expectations. 

•  Ensure  testing  to  meet  NATO  military  specifications  will  be  accomplished  by  NATO  or  entities  that 
can  demonstrate  the  needed  capabilities  and  security. 

B.2.2  Description  of  Needs 
B.2.2.1  Defining  the  Specifications 

Defining  the  specifications  for  medical  products  hinges  on  clearly  articulating  the  intended  use  and  indications 
for  use.  For  practical  and  regulatory  expediency,  only  medical  products  which  have  been  approved  by 
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appropriate  national  or  international  medical  product  regulatory  authority(ies)  shall  be  procured  by  NATO. 
NATO  (NAMSA)  will  determine  that  medical  products  are  registered,  i.e.,  have  a  Marketing  Authorization, 
or  other  such  document  from  an  appropriate  medical  product  regulatory  authority  (ies)  prior  to  contracting  to 
purchase  such  medical  equipment.  Further,  NAMSA  should  ensure  that,  in  the  case  of  medical  systems 
(e.g.,  telemedicine  systems,  evacuation  systems,  or  field  hospitals),  they  have  appropriately  been  tested  by 
ACT  or  another  appropriate  NATO  body  to  demonstrate  that  they  can  be  successfully  integrated  into  and/or 
be  used  within  the  NATO  multi-national  field  operational  environment. 

B.2.2.2  Emergency  Use  of  Approved  Medical  Products 

Emergency  use  of  approved  medical  products  for  non-approved  intended  use  and  indications  for  use  should 
meet  the  regulatory  expectations  of  the  appropriate  nation(s)  using  the  medical  product  or  medical  systems. 

B.2.2.3  Forecasting 

Forecasting  of  Needs  -  For  planning  purposes,  manufacturers  require  long-term  forecasts  for  products  and 
quantities,  preferably  for  six  to  24  months,  which  take  account  of  the  following:  safety  stocks,  shelf-life, 
cold-storage  and  cold  transportation  considerations,  as  needed,  and  warehousing  capacity  and  in-country 
logistics. 

B.2.2.4  Order  Quantity  and  Ordering  Interval 

It  is  important  to  establish  the  quantity  required  per  product  per  period,  and  ideally  plan  orders  in  advance. 

B.2.2.5  Delivery  Lead-Time 

It  is  vital  to  establish  when  the  products  are  required  to  arrive  in-country,  and  integrate  this  in  the  procurement 
plan.  Lead  times  can  be  up  substantial  for  some  products. 

B.2.2.6  Quality  Assurance  of  Products 

NATO  may  establish  specifications  that  address  military  expectations  for  the  performance  of  medical  products, 
such  as  field  ruggedness,  ability  to  interface  with  existing  medical  equipment;  ensure  interference  with  other 
equipment/medical  devices  does  not  occur,  etc.,  and  other  military  specific  needs: 

•  Pre-Qualification  Program  -  In  close  cooperation  with  national  regulatory  agencies  and  partner 
organizations,  a  NATO  Prequalification  Program  may  be  established  ad  hoc  that  seeks  interoperable, 
quality  medical  products,  including  drugs,  devices,  biologies,  and  other  medical  products  available  for 
NATO  forces  that  are  sensitive  to  the  regulatory  requirements  of  the  component  NATO  forces  and  the 
proposed  sites  of  medical  product  use.  This  may  be  optimally  achieved  through  its  evaluation  and 
inspection  activities,  and  by  building  multi-national  capacity  for  sustainable  manufacturing  and 
monitoring  of  quality  medical  products. 

B.2.2.7  Packaging 

Packaging  of  procured  medical  products  -  The  specific  packaging  materials  will  be  included  in  the  description 
of  needs,  and  shall  take  into  account  the  environments  where  the  medical  product(s)  will  be  utilized.  Such 
considerations  shall  include  how  the  environment  could  affect  the  stability,  safety,  efficacy,  performance, 
and  shelf  life  of  the  medical  products  in  their  deployed  situation.  Additionally,  specific  product  markings  as 
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may  be  required  by  various  STANAGS  will  be  specified  in  the  description  of  needs.  Testers  should  identify  if 
packaging  meets  identified  requirements: 

•  Pre-Shipment  Inspection:  The  need  for  pre-shipment  inspections  will  be  articulated  in  the  description 
of  needs. 

•  Consignee:  Full  consignee  details,  including  a  contact  person,  must  be  provided. 


B.3  MEDICAL  DEVICES  (INCLUDING  SYSTEMS)  AND  KITS 

To  support  its  program  goals  and  policies,  NATO,  through  NAMSA  or  through  the  member  nations, 
may  procure  medical  devices  that  meet  specific  program  requirements  and  can  be  supplied  at  competitive 
rates.  The  role  of  providing  medical  devices  is  essential  and  irreplaceable  in  the  delivery  of  healthcare. 

B.3.1  Medical  Devices  and  Kits 

Medical  devices  and  kits  cover  five  products  groups: 

•  Medical  equipment  (including  systems); 

•  Medical  renewable; 

•  Medical/hygiene  sets,  kits,  and  outfits; 

•  Laboratory  supplies;  and 

•  Diagnostic  test  kits. 

Each  of  these  product  groups  is  divided  into  several  sub-groups  which  lead  to  individual  items. 


B.4  DEFINITIONS  OF  MEDICAL  DEVICES 
B.4.1  Medical  Devices 

Medical  devices  are  any  instrument,  apparatus,  implement,  machine,  appliance,  implant,  in  vitro  regent  or 
calibrator,  software,  material  or  other  similar  or  related  article,  intended  by  the  manufacturer  to  be  used,  alone 
or  in  combination,  for  human  beings  for  one  or  more  of  the  specific  purpose(s)  of: 

•  Diagnosis,  prevention,  monitoring,  treatment  or  alleviation  of  disease. 

•  Diagnosis,  monitoring,  treatment,  alleviation  of  or  compensation  for  an  injury. 

•  Investigation,  replacement,  modification,  or  support  of  the  anatomy  or  of  a  physiological  process. 

•  Supporting  or  sustaining  life. 

•  Control  of  conception. 

•  Disinfecting  of  medical  devices. 

•  Providing  information  for  medical  purposes  by  means  of  in  vitro  examination  of  specimens  derived 
from  the  human  body  and  which  does  not  achieve  its  primary  intended  action  in  or  on  the  human  body 
by  pharmacological,  immunological  or  metabolic  means,  but  which  may  be  assisted  in  its  function  by 
such  means. 
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An  accessory  is  not  necessarily  considered  to  be  a  medical  device.  However,  when  an  accessory  is  intended 
specifically  by  its  manufacturer  to  be  used  together  with  the  “parent”  medical  device  to  enable  the  medical 
device  to  achieve  its  intended  purpose,  it  should  be  subject  to  the  same  procedures  that  apply  to  medical 
devices.  For  further  information  see  Global  Harmonization  Task  Force  (GHTF)  guidance  documents1. 

The  definition  of  a  device  for  in  vitro  examination  includes,  for  example,  reagents,  calibrator,  samples 
collection  devices,  control  materials,  and  related  instruments  or  apparatus.  The  information  provided  such  an 
in  vitro  diagnostic  device  may  be  for  diagnostic,  monitoring  or  compatibility  purposes.  In  some  jurisdictions, 
reagents  and  the  like  may  be  covered  by  separate  regulations. 

Products  which  are  considered  to  be  medical  in  some  jurisdictions  but  for  which  there  is  not  yet  an  international 
harmonized  approach,  are  not  covered  within  this  document,  e.g.,: 

•  Aids  for  disabled/handicapped  people. 

•  Devices  for  the  treatment/diagnosis  of  diseases  and  injuries  in  animals. 

•  Spare  parts  for  medical  devices. 

•  Devices  incorporating  animal  and  human  tissues  which  may  meet  the  requirements  of  the  above 
definitions  but  which  may  be  subject  to  different  controls. 


B.5  REGULATION 

If  a  decision  is  made  to  procure  a  medical  product  or  device  that  is  not  FDA  or  EC  approved,  then  emerging 
Global  Harmonization  Task  Force  (GHTF)  guidelines  may  provide  assistance.  Government  authorities  of  the 
country  in  which  a  product  is  manufactured  or  imported  usually  enforce  regulations.  However,  many  countries 
do  not  yet  have  regulations  for  medical  devices,  or  these  regulations  are  sparse  or  fragmentary.  Although  it  is 
encouraging  that  some  countries  are  in  the  process  of  establishing  such  regulations,  a  proliferation  of  different 
national  regulations  could  hinder  access  to  technology  and  should  be  avoided. 

The  GHTF2  is  a  voluntary  group  of  representatives  from  national  medical  devices  regulatory  authorities  and 
the  regulated  industry.  Since  its  inception,  the  GHTF  has  been  comprised  of  five  founding  members  grouped 
into  three  geographic  areas.  The  five  founding  members  are  the  European  Union,  Japan,  Australia,  Canada 
and  the  United  States  of  America.  The  three  geographic  areas  are  Europe,  Asia-Pacific  and  North  America, 
each  of  which  actively  regulates  medical  devices  using  their  own  unique  regulatory  framework. 

The  purpose  of  the  GHTF  is  to  encourage  convergence  in  regulatory  practices  related  to  ensuring  the  safety, 
effectiveness/performance  and  quality  of  medical  devices,  promoting  technological  innovation  and  facilitating 
international  trade.  This  is  primarily  accomplished  through  the  publication  and  dissemination  of  harmonized 
guidance  documents  on  basic  regulatory  practices. 

The  GHTF  also  serves  as  an  information  exchange  forum  through  which  countries  developing  medical  device 
regulatory  systems  can  benefit  from  the  experience  of  those  with  existing  systems  and/or  pattern  their 
practices  upon  those  of  GHTF  founding  members. 


1  GHTF  doc.  SGI/N029R1 1-  date:  February  25,  2002. 

2 

http://www.ghtf.org. 
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B.5.1  Standards  and  Regulatory  Requirements3 

The  means  to  ensure  the  safety  and  performance  of  the  medical  devices  include  standards  and  national 
regulations.  In  general,  standards  are  voluntary  while  regulations  are  mandatory.  However  regulations  can 
make  certain  standards  mandatory.  In  addition,  NATIONAL  or  MULTI-NATIONAL  purchasers  may  also 
have  their  own  specifications  that  a  product  must  fulfill. 

B.5.1. 1  Product  Standards 

Product  standards  describe  characteristics  and  performances  of  the  product. 

B.5.1.2  Quality  System  Standards 

Quality  system  standards  provide  the  essential  elements  that  a  manufacturer  should  have  in  place  to  ensure 
that  the  quality  of  products  manufactured  is  consistent. 

B.5.1.3  Types  of  Standards 

•  Product  standards; 

•  Procurement  specifications;  and 

•  Quality  system  standards. 

B.5.1.4  Elements  of  Regulatory  Requirements 

•  Conformity  with  product  standards; 

•  Consistency  in  product  quality; 

•  Packaging  and  labelling;  and 

•  Marketing  clearance. 

B.5.2  Proposed  Management  of  Medical  Device  Procurement 
B.5.2.1  NATO  Oversight  of  Medical  Procurement  Programs 

NATO  needs  to  designate  an  appropriate  NATO  entity  which  is  responsible  for  identifying  the  most 
appropriate  medical  devices  and  ensuring  that  NATO  offices  and  external  customers  can  purchase  quality 
medical  devices  at  an  affordable  price.  Normally,  such  decisions  would  follow  systematic  T&E  or  MTD 
studies  as  per  Chapters  4  and  5.  The  NTO,  responsible  for  system  testing  in  the  context  of  NATO  operations 
and  doctrine  would  not  normally  carry  out  this  type  of  pure  logistic  analysis  and  function,  which  could  most 
appropriately  be  carried  out  by  the  Medical  Branch  at  the  NATO  Maintenance  and  Supply  Agency  (NAMSA), 
after  the  requirements  are  doctrinally  evaluated  by  ACT. 


3 

Note:  There  are  different  types  of  standards  (e.g.:  private,  public,  organizational,  regional,  national  and  international  standards). 
As  health  care  products,  medical  devices  normally  follow  international  standards.  The  GHTF  aims  at  harmonizing  regulatory 
requirements  and  practices  based  upon  essential  principles  and  common  criteria.  At  this  time,  conformity  assessment  requirements 
and  other  regulatory  controls  assigned  to  each  class  of  devices  by  different  regulatory  authorities  have  yet  to  be  harmonized  and 
many  vary. 
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B.5.2.2  NAMSA  Procurement  Role 

As  NATO  develops  its  medical  procurement  role,  NAMSA  or  some  other  designated  logistic  agency  can 
provide  technical  support  to  national  or  organizational  procurers,  which  support  could  include: 

•  Assessing  and  analyzing  NATO  needs  in  the  context  of  health  systems,  policy  and  regulation,  and  the 
supply  system. 

•  Assessing  health  structures  regarding  supply  planning  for  various  types  of  medical  needs. 

•  Defining  customer  needs  for  products  in  the  context  of  the  environment  of  utilization,  policy  and 
legislation,  international  norms  and  regulation  and  promotion  of  protocols  for  the  evaluation  of 
product  use,  replenishment,  maintenance  and  quality. 

•  Assisting  appropriate  NATO  agencies  to  perform  appropriate  Testing  and  Evaluation  of  the  use  of 
medical  devices  and  systems  in  the  multi-national  and  field  setting,  as  recommended  in  other  parts  of 
this  document. 

•  Assessing  and  analyzing  sourcing  options  for  NATO  standard  and  non-standard  products. 

•  Assessing  the  supply  chain  and  identifying  needs  to  optimize  the  supply  function. 

•  Identifying  priorities/constraints  in  coordination  with  different  partners  involved  in  implementing  a 
project  in  order  to  address  customer  needs  effectively. 

•  Providing  support  for  training. 

B.5.2.3  Proposed  NATO  Procurement  Criteria  for  Medical  Devices  and  Kits 

•  Purchase  only  from  appropriately  licensed  entities. 

•  Purchase  only  from  manufacturers  with  a  proven  record  of  competence,  knowledge  and  experience. 

•  Purchase  only  from  suppliers  of  medical  devices  certified  in  accordance  with  ISO  9001/EN46001  or 
ISO  9002/EN  46002. 

•  Purchase  only  from  manufacturers  who  conform  to  Good  Manufacturing  Practices  (GMP)  and  quality 
management  system  /  design  control  guidelines  and  which  are  regularly  inspected  by  the  appropriate 
national  regulatory  authorities. 

•  Purchase  only  from  those  vendor  or  producers  that  have  good  manufacturing  practice  /  quality 
management  system  /  design  control  inspections  performed  by  NATO’s  own  qualified  GMP  inspector, 
by  a  Foreign  Regulatory  GMP  Inspectorate  or  by  independent  consultants  with  a  long  experience  in  this 
area.  Suppliers  are  normally  inspected  every  2  to  4  years,  depending  on  the  regulatory  enviromnent  in 
the  country  of  origin. 

•  A  Marketing  Authorization  for  the  product  in  the  market  of  the  country  of  origin  is  generally  required 
for  medical  devices.  Products  may  also  meet  the  essential  requirements  described  in  the  EEC 
Directive:  Council  Directive  93/42/EEC  of  14/06/1993  (CE  Marking). 

•  An  independent  laboratory  analysis  of  randomly  picked  samples  of  the  products  delivered  to  NATO 
may  be  conducted. 
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•  Testing  to  meet  NATO  specific  military  specifications  should  be  accomplished  by  NATO  or  entities 
that  can  demonstrate  the  needed  capabilities  and  security. 

•  The  suitability  of  packing  and  labeling  should  be  subject  to  both  technical  and  quality  checks  before 
purchasing. 

•  The  minimum  shelf-life  remaining  of  medical  products  supplied  to  NATO  should  be  24  months  or  as 
indicated  in  the  description  of  need. 

•  A  recall  system  is  in  place  for  quality  issues. 

•  Establish  technical,  quality  assurance,  and  procurement  processes  to  ensure  that  both  suppliers  and 
the  medical  devices  they  offer  are  in  accordance  with  NATO’s  expectations. 

B.5.2.4  Technical  and  Quality  Assurance  Processes 

Technical  and  quality  assurance  processes  for  medical  devices  cover: 

•  Supplier  quality  assurance  system:  good  manufacturing  and  distribution  practice; 

•  Supplier  inspection  of  manufacturing  site; 

•  Product  conformity  to  international  standards; 

•  Product  specifications; 

•  Product  documentation; 

•  Product  shelf  life  (when  applicable); 

•  Certificate  of  sterilization  (when  applicable); 

•  Product  packaging  and  labelling;  and 

•  Product  marketing  authorizations. 

B.5.2.5  Specific  Requirements  for  Equipment 

Specific  requirements  for  equipment  include: 

•  Product  conformity  with  final  destination  of  goods; 

•  Supplier  services:  installation,  training  on  site;  and 

•  Supplier  warranty  and  after-sale  service. 

B.5.3  Drugs/Pharmaceuticals,  Biologies,  Vaccines,  Combination  Products,  and  Medical 
Foods 

B.5.3. 1  Definitions 

•  Drugs/Pharmaceuticals:  Drugs  are  defined  by  their  intended  use,  as  “articles  intended  for  use  in  the 
diagnosis,  cure,  mitigation,  treatment,  or  prevention  of  disease,  and  articles  (other  than  food)  intended  to 
affect  the  structure  or  any  function  of  the  body  of  man  or  other  animals”. 

•  Biologies  and  Vaccines:  A  biological  product  is  defined  as  “a  virus,  therapeutic  serum,  toxin,  anti-toxin, 
vaccine,  blood,  blood  component  or  derivative,  allergenic  product,  or  analogous  product,  applicable  to  the 
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prevention,  treatment  or  cure  of  a  disease  or  condition  of  human  beings”.  Biological  products  include 
viruses,  therapeutic  sera,  toxins  and  anti-toxins,  vaccines,  blood,  blood  components  or  derivatives,  allergenic 
products,  and  any  analogous  products,  used  for  treating  disease. 

•  Combination  Product:  The  term  combination  product  includes: 

1)  A  product  comprised  of  two  or  more  regulated  components,  i.e.,  drug/device,  biologic/device, 
drug/biologic,  or  drug/device/biologic,  that  are  physically,  chemically,  or  otherwise  combined  or 
mixed  and  produced  as  a  single  entity. 

2)  Two  or  more  separate  products  packaged  together  in  a  single  package  or  as  a  unit  and  comprised  of 
drug  and  device  products,  device  and  biological  products,  or  biological  and  drug  products. 

3)  A  drug,  device,  or  biological  product  packaged  separately  that  according  to  its  investigational  plan  or 
proposed  labeling  is  intended  for  use  only  with  an  approved  individually  specified  drug,  device, 
or  biological  product  where  both  are  required  to  achieve  the  intended  use,  indication,  or  effect  and  where 
upon  approval  of  the  proposed  product  the  labeling  of  the  approved  product  would  need  to  be  changed, 
e.g.,  to  reflect  a  change  in  intended  use,  dosage  form,  strength,  route  of  administration,  or  significant 
change  in  dose. 

•  Medical  Foods:  A  medical  food  is  “a  food  which  is  formulated  to  be  consumed  or  administered  entirely 
under  the  supervision  of  a  physician  and  which  is  intended  for  the  specific  dietary  management  of  a 
disease  or  condition  for  which  distinctive  nutritional  requirements,  based  on  recognized  scientific 
principles,  are  established  by  medical  evaluation”.  Medical  foods  are  distinguished  from  the  broader 
category  of  foods  for  special  dietary  use  and  from  foods  that  make  health  claims  by  the  requirement  that 
medical  foods  be  intended  to  meet  distinctive  nutritional  requirements  of  a  disease  or  condition,  used 
under  medical  supervision  and  intended  for  the  specific  dietary  management  of  a  disease  or  condition. 
The  term  “medical  foods”  does  not  pertain  to  all  foods  fed  to  sick  patients.  Medical  foods  are  foods  that 
are  specially  formulated  and  processed  (as  opposed  to  a  naturally  occurring  foodstuff  used  in  a  natural 
state)  for  the  patient  who  is  seriously  ill  or  who  requires  the  product  as  a  major  treatment  modality. 
In  general,  to  be  considered  a  medical  food,  a  product  must,  at  a  minimum,  meet  the  following  criteria: 
the  product  must  be  a  food  for  oral  or  tube  feeding;  the  product  must  be  labeled  for  the  dietary 
management  of  a  specific  medical  disorder,  disease,  or  condition  for  which  there  are  distinctive  nutritional 
requirements;  and  the  product  must  be  intended  to  be  used  under  medical  supervision. 

B.5.3.2  NATO  (and  NATO  Nations’)  Procurement  Criteria  for  Medical  Products  (Pharmaceuticals, 
Biologies,  Vaccines  and  Medical  Foods) 

The  quality  of  medical  products  is  a  major  concern  for  NATO.  As  a  result  the  following  policies  are 

recommended: 

•  Purchase  only  from  appropriately  licensed  Entities. 

•  Purchase  only  from  manufacturers  with  a  proven  record  of  competence,  knowledge  and  experience. 

•  Ensure  manufacturers  conform  to  Good  Manufacturing  Practices  (GMP)  guidelines  and  be  regularly 
inspected  by  national  regulatory  authorities.  Good  Manufacturing  Practice  inspections  are  performed 
by  a  NATO  qualified  or  designated  GMP  inspector,  by  a  Foreign  Regulatory  GMP  Inspectorate  or  by 
independent  consultants  with  a  long  experience  in  this  area.  Suppliers  are  normally  inspected  every 
2  to  4  years,  depending  on  the  regulatory  environment  in  the  country  of  origin. 

•  A  Marketing  Authorization  for  the  product  in  the  market  of  the  country  of  origin  is  generally  required 
for  medical  products  (Pharmaceuticals,  Biologies,  Vaccines  and  Medical  Foods). 
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•  A  Certificate  of  Analysis  can  accompany  all  medical  products  (Pharmaceuticals,  Biologies,  Vaccines 
and  Medical  Foods)  for  each  batch  supplied  as  appropriate. 

•  An  independent  laboratory  analysis  of  randomly  picked  samples  of  the  products  delivered  to  NATO 
may  be  conducted. 

•  Testing  to  meet  NATO  specific  military  specifications  will  be  accomplished  by  NATO  or  entities  that 
can  demonstrate  the  needed  capabilities  and  security. 

•  The  suitability  of  packing  and  labeling  is  subject  to  both  technical  and  quality  checks  before 
purchasing. 

•  The  minimum  shelf-life  remaining  of  medical  products  (Pharmaceuticals,  Biologies,  Vaccines  and 
Medical  Foods)  supplied  to  NATO  is  24  months  or  as  indicated  in  the  description  of  need. 

•  A  recall  system  is  in  place  for  quality  issues. 

B.5.3.3  Pharmaceuticals,  Biologies,  Vaccines  and  Medical  Foods  Procured  by  NATO 

Pharmaceuticals,  biologies,  vaccines  and  medical  foods  procured  by  NATO  must  be  manufactured  and  conform 
to  the  latest  edition  of  British,  United  States,  European  or  International  Pharmacopoeia,  if  such  a  monograph 
exists.  For  non-pharmacopoeia  products,  the  finished  product  specifications  from  the  description  of  needs  are 
used.  Raw  materials  used  in  manufacturing  must  be  of  good  quality  and  from  approved  sources  only  and 
conform  to  the  standards  in  the  latest  edition  of  British  Pharmacopoeia,  United  States  Pharmacopoeia,  European 
Pharmacopoeia  or  International  Pharmacopoeia,  whenever  applicable.  Pharmaceutical  containers  must  conform 
to  the  latest  pharmacopoeia,  whenever  applicable.  Packaging  should  be  suitable  for  delivery  and  use  in  countries 
with  adverse  climatic  and  storage  conditions  and  consistent  with  military  specifications  and  be  suitable  for 
shipment,  storage  and  use  worldwide  at  temperatures  and  humidity  specified  in  the  description  of  needs. 
The  containers  should  normally  be  tamper-proof  and  the  size  proportional  to  the  contents  with  the  addition  of 
appropriate  padding  to  prevent  damage  to  the  product  during  shipment. 


This  Annex  was  prepared  by: 

Richard  A.  Baldwin 

AF  SGR  -  Modernization  >  SGRC 

Food  and  Drug  Administration  (FDA)  /  Subject  Matter  Expert  (SME) 

Suite  1206  Sky  Three 
5201  Leesburg  Pike 
Falls  Church,  Virginia  22041 

703-575-2077  (Office) 

301-370-0235  (Mobile) 

Email:  richard.baldwin-02.ctr@pentagon.af.mil 

Mr.  Baldwin  has  33+  years  of  FDA  experience,  both  field  (last  assignment  Pacific  Regional  Director  -  SES) 
and  HQ  (last  assignment  Senior  Advisor  for  Science  -  SES).  Post  FDA,  Richard  served  as  Senior  Director, 
Quality  Audits  at  the  American  Red  Cross  and  Center  of  Excellence  Director,  Global  Regulatory  Affairs  at 
Baxter  Healthcare.  Richard  presently  serves  as  the  AF  SG  FDA  Subject  Matter  Expert. 
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Annex  C  -  REVIEWING  CAPABILITIES  BASED 
REQUIREMENTS  DOCUMENTS 


ORGANIZATION 


C.l  TESTER  DOCUMENT  REVIEW  RESPONSIBILITIES 

In  developing  a  test  plan,  the  plan  developer  needs  to  ensure  that  the  following  items  are  addressed: 

•  Capability  Discussion  -  Review  this  section  to  make  sure  it  is  an  overarching  discussion  that 
encompasses  Key  Performance  Parameters  (KPP),  thresholds  and  objectives. 

•  Concept  of  Operations  Summary  -  Review  this  section  to  make  sure  the  concepts  discussed  are 
directly  tied  to  requirements. 

•  Threat  Summary  -  Make  sure  that  any  discussion  regarding  Threats  to  he  Countered  addresses 
threats  to  the  proposed  system  that  the  system  is  expected  to  counter. 

•  Environmental  Impact  -  Make  sure  that  member  nation  guidance  regarding  environmental  concerns 
to  include  hazardous  conditions  is  adhered  to. 

•  System  Capabilities  Required  for  the  Current  Increment  -  Increases  requirement  by  adding  KSAs 
(key  system  attributes): 

•  If  this  section  contains  an  effectiveness  requirement  that  includes  reliability  it  should  be 
commented  on.  T&E  results  can  be  compromised  by  allowing  a  low  reliability  to  be  compensated 
for  by  high  effectiveness.  This  can  cause  problems  down  the  road  leading  to  additional  testing. 

•  Review  this  section  making  sure  only  a  minimum  number  of  requirements  are  KPP. 

•  Attributes  should  be  validated  for  testability. 

•  Know  the  difference  between  KPP/KSA  (see  definitions  in  Glossary). 

•  Performance  attributes  apply  only  to  a  single  increment  so  ensure  testing  can  be  accomplished  on 
current  planned  increment. 

•  Schedule  and  Initial  Operational  Capability  (IOC)/Full  Operational  Capability  (FOC)  Definitions 

-  Make  sure  IOT&E/FOT&E  is  discussed  here  if  appropriate. 

•  Other  System  Attributes  -  Check  this  section  for  additional  testability  of  system  attributes. 
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Annex  D  -  OPERATIONAL  TEST:  TEST  CONCEPT  (OT  TC) 

(To  be  used  as  an  example  of  a  Usable  Plan  - 
Not  to  be  Considered  Directive  in  Nature) 


CONTENTS 


SECTION 

EXECUTIVE  SUMMARY 

FIGURES 

TABLES 

ABBREVIATIONS  AND  ACRONYMS 

I  INTRODUCTION 
1.0  GENERAL 

1 . 1  SYSTEM  INFORMATION 

1.1.1  Background 

1.1.2  Description 

1.2  OPERATIONAL  ENVIRONMENT 

1.2.1  Threat  Summary 

1.2.2  Operational  Concept 

1.2.3  Maintenance  Concept 

1.2.4  Training  Concept 

1 . 3  PROGRAM  STRUCTURE 

II  OT&E  OUTLINE 

1 .0  CRITICAL  OPERATIONAL  ISSUES  AND  OBJECTIVES 

2. 1  SCOPE  AND  TEST  CONCEPT 

2.2  PLANNING  CONSIDERATIONS  AND  LIMITATIONS 

2.2. 1  Planning  Considerations 

2.2.2  Integrated  Testing  Considerations 

2.2.3  Limitations 

2.2.4  Estimated  Cost 

2.3  CONTRACTOR  INVOLVEMENT 

2.4  OT&E  SCHEDULE  AND  READINESS  REQUIREMENTS 

III  METHODOLOGY 
3.0  GENERAL 

3.0.1  COI  Summary 

3.0.2  COI  and  MOE/MOP  Matrix 

3.0.3  Integrated  Test  Matrix 
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3.1  COI-1 

3.1.1  Scope 

3.1.2  MOEs,  MOPs,  and  Evaluation  Criteria 

3.1.3  Mission  Scenarios 

3.1.4  Method(s)  of  Evaluation 

3.2  COI  -  2 

3.3  COI  -  3 

3.X  SURVIVABILITY  ASSESSMENT 

IV  ADMINISTRATION 

4.0  TEST  MANAGEMENT 

4.1  TASKING 

4.2  TRAINING  REQUIREMENTS 

4.3  SAFETY 

4.4  ENVIRONMENTAL  IMPACTS 

4.5  SOURCE  SELECTION  INFORMATION 

4.6  SECURITY 

V  REPORTING 
5.0  REPORTS 

5.1  BRIEFINGS 

5.2  DEFICIENCY  REPORTING 

OPTIONAL  SUPPLEMENTS 

A  INTELLIGENCE  AND  THREAT  ASSESSMENT  (CLASSIFIED) 
B  OPERATIONS  SECURITY 
C  CLASSIFIED  EVALUATION  CRITERIA 
D  DETAILED  SYSTEM  DESCRIPTION 
E  SOFTWARE  EVALUATION 
F  HUMAN  FACTORS 
G  WEATHER 

H  MODELING  &  SIMULATION 

GLOSSARY  OF  TERMS 

REFERENCES 

DISTRIBUTION 
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Annex  E  -  DRAFT  TEMPLATE:  OPERATIONAL 
TEST  FINAL  REPORT 

(To  be  used  as  an  Example  of  a  Final  Report  - 
Not  to  be  Considered  Directive  in  Nature) 

CONTENTS 


SECTION 

EXECUTIVE  SUMMARY 

FIGURES 

TABLES 

ABBREVIATIONS  AND  ACRONYMS 

I  PURPOSE  AND  BACKGROUND 
1.0  OT&E  PURPOSE 

1 . 1  AUTHORIZING  DIRECTIVES 

1.2  OT&E  BACKGROUND 

1.3  DESCRIPTION  OF  SYSTEM  TESTED 

1 .4  TEST  FORCE,  LOCATION,  DATES 

1 . 5  CL  AS  SIFICATION  STATEMENT 

II  OT&E  DESCRIPTION 

2.0  CRITICAL  OPERATIONAL  ISSUES 

2. 1  SCOPE  AND  METHOD  OF  ACCOMPLISHMENT 

2.2  PLANNING  CONSIDERATIONS  AND  LIMITING  FACTORS 

2.3  CONTRACTOR  INVOLVEMENT 

III  OPERATIONAL  EFFECTIVENESS  AND  SUITABILITY 
3.0  SUMMARY 

3.1  COI-1 

3.1.1  Method 

3.1.2  Results  and  Conclusions 

3.1.3  Recommendations 

3.2  COI  -  2 

3.3  COI  -  3 

(List  additional  COIs  in  same  sequence  as  the  test  plan) 

3.X  SURVIVABILITY  ASSESSMENT 

3.Y  ENVIRONMENTAL  IMPACTS 

3.Z  OPERATIONAL  IMPACT  ASSESSMENTS 
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IV  DEFICIENCY  REPORTS 

4.0  DEFICIENCY  REPORT  STATUS 

4.1  IMPACT  SUMMARY 

4.2  PRIORITIZED  DEFICIENCY  REPORTS 

V  SUMMARY  OF  CONCLUSIONS  AND  RECOMMENDATIONS 

GLOSSARY 

REFERENCES 

ATTACHMENTS 

A  EXECUTIVE  SUMMARY  OF  PREVIOUS  OT&E 
B-Z  AS  NEEDED 
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Annex  F  -  REFERENCES  AND  SUPPORTING  INFORMATION 
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Region,  Edmonton,  Canada,  June  2006. 

Alberta  Heritage  Foundation  for  Medical  Research,  Risk  Management  for  Health  Technology  Assessment 
Programs,  Edmonton,  Canada.  May  2005. 

Canadian  Agency  for  Drugs  and  Technologies  in  Health,  Guidelines  for  the  Economic  Evaluation  of  Health 
Technologies,  Ottawa,  Canada,  2006. 
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I.  BACKGROUND  AND  JUSTIFICATION 

Advanced  medical  technologies  (often  called  telemedicine)  are  widely  seen  as  having  potential  benefit  to 
patients,  clinicians,  and  commanders  in  multi-national  operations  in  a  field  setting.  Adoption  of  any  particular 
technology  or  telemedicine  modality  for  Alliance  support  is  hindered  by  the  lack  of  any  standardized  NATO 
assessment  mechanism  which  can  evaluate  and  assess  new  medical  technologies  for  technical  maturity, 
usability  (in  the  classic  human  factors  sense),  and  benefits  to  the  patient,  the  clinician,  and  the  operational 
commander.  Development  of  such  an  evaluation  mechanism  could  lead  to  increased  deployment  of  such 
advanced  medical  technologies,  as  well  as  increased  interoperability. 

II.  OBJECTIVE(S) 

•  To  assemble  a  database  of  tried/proposed  evaluation  methodologies  for  new  advanced  medical  technology; 

•  To  use  the  best  of  these  to  produce  a  single  standardized  NATO  test  and  evaluation  methodology  for 
this  type  of  equipment;  to  test  the  new  methodology  during  a  Cooperative  Technology  Demonstration 
(CDT)  -  in  conjunction  with  a  NATO  exercise; 

•  To  modify  the  proposed  methodology  as  needed;  and 

•  To  publish  the  standardized  methodology  as  a  NATO  standardization  document. 
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III.  TOPICS  TO  BE  COVERED 

•  Historical  review  of  advanced  medical  technology  test  and  evaluation  procedures; 

•  Operational  issues; 

•  Functional  issues; 

•  Human  factors  usability  issues; 

•  NATO  requirements; 

•  Operational  testing;  and 

•  Standardization. 

IV.  DELIVERABLE  AND/OR  END  PRODUCT 

•  Database  of  current/proposed  test  and  evaluation  procedures; 

•  Standardized  NATO  test  and  evaluation  procedure;  and 

•  Standardization  document. 


V.  TECHNICAL  TEAM  LEADER  AND  LEAD  NATION 

Dr.  David  Lam  (USA). 


VI.  NATIONS  WILLING  TO  PARTICIPATE 

BUL,  CAN,  CZE,  FIN,  GBR,  ITA,  NOR,  USA. 

VII.  NATIONAL  AND/OR  NATO  RESOURCES  NEEDED 

Some  nations  already  have  national  programs  to  develop  and  test  such  advanced  medical  technology.  The  Task 
Group  (TG)  members  need  to  obtain  permission  for  the  release  of  national  test  protocols  and  experience  to  the 
TG.  During  the  CDT  phase,  nations  will  need  to  be  willing  to  loan  such  medical  devices/equipment  for  field- 
testing  in  accordance  with  the  developed  test  methodology.  Each  nation  will  be  responsible  for  its  own  travel 
funding.  ACO  and  ACT  Medical  Advisors  will  need  to  provide  input  to  the  TG  on  operational  requirements  for 
such  advanced  medical  technology,  as  well  as  on  development  of  the  scenario  for  the  CDT.  The  CDT  will  have 
to  be  invited  by  ACO/ACT  to  integrate  the  CDT  into  a  NATO  field  exercise,  and  the  Exercise  DISTAFF  will 
need  to  support  such  integration  logistically  and  operationally. 


VIII.  RTA  RESOURCES  NEEDED 

•  Normal  support  for  RTO  Task  Group. 

•  Hosting  of  first  meeting  at  RTA  HQ. 
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